Xen devel - Oct 2010 - [PATCH 00/13] Nested Virtualization: Overview

Hi!

This patch series brings Nested Virtualization to Xen.
This is the fifth patch series. Improvements to the
previous patch submission:

- Incorporated feedback from discussion with Tim and Eddie.
- Reworked common data structure and function hooks (patch #2 and #3)
   Adjusted all patches affected by this.
- Make use of hvm_map_guest_frame() introduced in c/s 22228:02e199c96ece
- Removed workaround in hap-on-hap patch thanks to c/s 22215:c8331262efe6
- Some bugfixes


The patch series:

patch 01: add nestedhvm guest config option to the tools.
                  This is the only one patch touching the tools
patch 02: Add data structures for nested virtualization.
patch 03: add nestedhvm function hooks.
patch 04: The heart of nested virtualization.
patch 05: Allow switch to paged real mode during vmrun emulation.
                  Emulate cr0 and cr4 when guest does not intercept them
                  (i.e. Hyper-V/Windows7, KVM)
patch 06: When injecting an exception into nested guest, inject
                  #VMEXIT into the guest if intercepted.
patch 07: Allow guest to enable SVM in EFER only on AMD.
patch 08: Handle interrupts (generic part).
patch 09: SVM specific implementation for nested virtualization.
patch 10: Handle interrupts (SVM specific).
patch 11: The piece of code that effectively turns on nested virtualization.
patch 12: Move dirty_vram from struct hvm_domain to struct p2m_domain.
                  This change is the first part from a larger not-yet-ready
                  change where the vram and log_dirty tracking is teached
                  to work on per p2m.
patch 13: Handle nested pagefault to enable hap-on-hap and handle
                  nested guest page-table-walks to emulate instructions
                  the guest does not intercept (i.e. WBINVD with Windows 7).


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Christoph Egger wrote:
> Hi!
> 
> This patch series brings Nested Virtualization to Xen.
> This is the fifth patch series. Improvements to the
> previous patch submission:
> 
It is a step forward progress. At least those obvious SVM specific stuff is
moved back to SVM tree, although there are still something left which is not
good but maybe not that critical (like
-    hvm_intblk_nmi_iret   /* NMI blocked until IRET */
+    hvm_intblk_nmi_iret,  /* NMI blocked until IRET */
+    hvm_intblk_svm_gif,   /* GIF cleared */
).

However the majority of my comments are still not addressed
(http://www.mailinglistarchive.com/html/xen-devel@lists.xensource.com/2010-09/msg01078.html).

I can re-comments here:

+struct nestedhvm {
+    bool_t nh_guestmode; /* vcpu in guestmode? */
+    void *nh_vmcx; /* l1 guest virtual VMCB/VMCS */
+
+    /* address of l1 guest virtual VMCB/VMCS, needed for VMEXIT */
+    uint64_t nh_vmaddr;
+
+    void *nh_arch; /* SVM/VMX specific data */
+    size_t nh_arch_size;

I hate the pointer+size style. Rather I prefer union for SVM/VMX data structure.

Once this is followed, the API
nestedhvm_vcpu_initialise/nestedhvm_vcpu_reset/nestedhvm_vcpu_destroy can be
back to wrapper only if not completely removed.


+    /* Cache guest cr3/host cr3 the guest sets up for the nested guest.
+     * Used by Shadow-on-Shadow and Nested-on-Nested.
+     * nh_vm_guestcr3: in l2 guest physical address space and points to
+     *     the l2 guest page table
+     * nh_vm_hostcr3: in l1 guest physical address space and points to
+     *     the l1 guest nested page table
+     */
+    uint64_t nh_vm_guestcr3, nh_vm_hostcr3;
+    uint32_t nh_guest_asid;

Duplicating data instance (Caching) is really not a good solution to me. I
prefer using a wrapper API for that as my proposal sent to you in private mail.
Caching really doesn''t bring additional value here.


+    /* Only meaningful when forcevmexit flag is set */
+    struct {
+        uint64_t exitcode;  /* generic exitcode */
+        uint64_t exitinfo1; /* additional information to the exitcode */
+        uint64_t exitinfo2; /* additional information to the exitcode */
+    } nh_forcevmexit;
+    union {
+        uint32_t bytes;
+        struct {
+            uint32_t forcevmexit : 1;
+            uint32_t use_native_exitcode : 1;
+            uint32_t vmentry : 1;   /* true during vmentry/vmexit emulation */
+            uint32_t reserved : 29;
+        } fields;
+    } nh_hostflags;

New namespace for VM exit info and entry control.

I am not convinced by the value of this new namespace comparing with soultion
that demux in each arch while call 1st level helper API in common code. The only
different result is that we will pay with one API: nestedhvm_vcpu_vmexit, but we
gain with removed conversion to/from between new & old namespace APIs. I
strongly prefer the demux + 1st level helper solution.

+int
+nestedhvm_vcpu_vmentry(struct vcpu *v, struct cpu_user_regs *regs,
+	uint64_t vmaddr, unsigned int inst_len)
+{
+	int ret;
+	struct nestedhvm *hvm = &vcpu_nestedhvm(v);
+
+	hvm->nh_hostflags.fields.vmentry = 1;
+
+	ret = nestedhvm_vcpu_state_validate(v, vmaddr);
+	if (ret) {
+		gdprintk(XENLOG_ERR,
+			"nestedhvm_vcpu_state_validate failed, injecting 0x%x\n",
+			ret);
+		hvm_inject_exception(ret, HVM_DELIVER_NO_ERROR_CODE, 0);
+		return ret;
+	}
+
+	/* Save vmaddr. Needed for VMEXIT */
+	hvm->nh_vmaddr = vmaddr;
+
+	/* get nested vm */
+	ASSERT(hvm->nh_vmcx == NULL);
+	hvm->nh_vmcx = hvm_map_guest_frame_ro(vmaddr >> PAGE_SHIFT);
+	if (hvm->nh_vmcx == NULL) {
+		gdprintk(XENLOG_ERR,
+			"hvm_map_guest_frame_ro failed, injecting #GP\n");
+		hvm_inject_exception(TRAP_gp_fault,
+			HVM_DELIVER_NO_ERROR_CODE, 0);
+		hvm->nh_hostflags.fields.vmentry = 0;
+		return TRAP_gp_fault;
+	}
+
+	/* save host state */
+	ret = nhvm_vcpu_vmentry(v, regs, inst_len);
+	if (ret) {
+		gdprintk(XENLOG_ERR,
+			"nhvm_vcpu_vmentry failed, injecting #UD\n");
+		hvm_inject_exception(TRAP_invalid_op,
+			HVM_DELIVER_NO_ERROR_CODE, 0);
+		hvm->nh_hostflags.fields.vmentry = 0;
+		hvm_unmap_guest_frame(hvm->nh_vmcx);
+		hvm->nh_vmcx = NULL;
+		return ret;
+	}
+
+	hvm_unmap_guest_frame(hvm->nh_vmcx);
+	hvm->nh_vmcx = NULL;
+
+	/* Switch vcpu to guest mode.
+	 */
+	nestedhvm_vcpu_enter_guestmode(v);
+
+	hvm->nh_hostflags.fields.vmentry = 0;
+	return 0;
+}

You must not read VMX code yet or didn''t understand the reason why VMX
can;t do this. Appearantly this common code doesn''t apply to VMX and
have to be moved to arch specific stuff.
VMX can only switch the context (L1 or L2 guest) at certain point (now defered
to last step before resume) because only one VMCS can be accessed in one
context.

Thx, Eddie



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Eddie,

I think I have found a way to address most if not all of your concerns.

As a first step I have:
- made SVM''s VMRUN emulation SVM-specific
- turned nh_arch into a union
- moved msr permission maps into SVM
- made nested pagefault vmexit injection SVM-specific
- removed nh_arch_size
- renamed nh_vmaddr to nh_vmcxaddr
- moved hvm_intblk_svm_gif into SVM

Next I will:
- make fields like nh_vm_guestcr3, nh_vm_hostcr3, and nh_guest_asid 
SVM-specific (we need to cache those values for architectural reasons) and 
provide a wrapper for where these values may be needed by generic code
- move nestedhvm_vcpu_interrupt() into SVM. This will remove the generic 
exitcode field from ''struct nestedhvm'' and will allow me to
move
the ''exitinfo1'' and ''exitinfo2'' fields into
SVM.
- make SVM''s VMEXIT emulation SVM-specific. That should do away with
state
validation and mapping issues and end the confusion around nestedhvm_vmexit, 
nestedhvm_vcpu_vmexit, nhvm_vcpu_vmexit, too.
- address other outstanding terminology issues such as nhvm, struct nvcpu

It will probably take a few days to get this straightened out, so stay tuned.

Christoph


On Monday 18 October 2010 03:30:10 Dong, Eddie wrote:
> Christoph Egger wrote:
> > Hi!
> >
> > This patch series brings Nested Virtualization to Xen.
> > This is the fifth patch series. Improvements to the
> > previous patch submission:
>
> It is a step forward progress. At least those obvious SVM specific stuff is
> moved back to SVM tree, although there are still something left which is
> not good but maybe not that critical (like -    hvm_intblk_nmi_iret   /*
> NMI blocked until IRET */
> +    hvm_intblk_nmi_iret,  /* NMI blocked until IRET */
> +    hvm_intblk_svm_gif,   /* GIF cleared */
> ).
>
> However the majority of my comments are still not addressed
> (http://www.mailinglistarchive.com/html/xen-devel@lists.xensource.com/2010-
>09/msg01078.html).
>
> I can re-comments here:
>
> +struct nestedhvm {
> +    bool_t nh_guestmode; /* vcpu in guestmode? */
> +    void *nh_vmcx; /* l1 guest virtual VMCB/VMCS */
> +
> +    /* address of l1 guest virtual VMCB/VMCS, needed for VMEXIT */
> +    uint64_t nh_vmaddr;
> +
> +    void *nh_arch; /* SVM/VMX specific data */
> +    size_t nh_arch_size;
>
> I hate the pointer+size style. Rather I prefer union for SVM/VMX data
> structure.
>
> Once this is followed, the API
> nestedhvm_vcpu_initialise/nestedhvm_vcpu_reset/nestedhvm_vcpu_destroy can
> be back to wrapper only if not completely removed.
>
>
> +    /* Cache guest cr3/host cr3 the guest sets up for the nested guest.
> +     * Used by Shadow-on-Shadow and Nested-on-Nested.
> +     * nh_vm_guestcr3: in l2 guest physical address space and points to
> +     *     the l2 guest page table
> +     * nh_vm_hostcr3: in l1 guest physical address space and points to
> +     *     the l1 guest nested page table
> +     */
> +    uint64_t nh_vm_guestcr3, nh_vm_hostcr3;
> +    uint32_t nh_guest_asid;
>
> Duplicating data instance (Caching) is really not a good solution to me. I
> prefer using a wrapper API for that as my proposal sent to you in private
> mail. Caching really doesn''t bring additional value here.
>
>
> +    /* Only meaningful when forcevmexit flag is set */
> +    struct {
> +        uint64_t exitcode;  /* generic exitcode */
> +        uint64_t exitinfo1; /* additional information to the exitcode */
> +        uint64_t exitinfo2; /* additional information to the exitcode */
> +    } nh_forcevmexit;
> +    union {
> +        uint32_t bytes;
> +        struct {
> +            uint32_t forcevmexit : 1;
> +            uint32_t use_native_exitcode : 1;
> +            uint32_t vmentry : 1;   /* true during vmentry/vmexit
> emulation */ +            uint32_t reserved : 29;
> +        } fields;
> +    } nh_hostflags;
>
> New namespace for VM exit info and entry control.
>
> I am not convinced by the value of this new namespace comparing with
> soultion that demux in each arch while call 1st level helper API in common
> code. The only different result is that we will pay with one API:
> nestedhvm_vcpu_vmexit, but we gain with removed conversion to/from between
> new & old namespace APIs. I strongly prefer the demux + 1st level
helper
> solution.
>
> +int
> +nestedhvm_vcpu_vmentry(struct vcpu *v, struct cpu_user_regs *regs,
> +	uint64_t vmaddr, unsigned int inst_len)
> +{
> +	int ret;
> +	struct nestedhvm *hvm = &vcpu_nestedhvm(v);
> +
> +	hvm->nh_hostflags.fields.vmentry = 1;
> +
> +	ret = nestedhvm_vcpu_state_validate(v, vmaddr);
> +	if (ret) {
> +		gdprintk(XENLOG_ERR,
> +			"nestedhvm_vcpu_state_validate failed, injecting 0x%x\n",
> +			ret);
> +		hvm_inject_exception(ret, HVM_DELIVER_NO_ERROR_CODE, 0);
> +		return ret;
> +	}
> +
> +	/* Save vmaddr. Needed for VMEXIT */
> +	hvm->nh_vmaddr = vmaddr;
> +
> +	/* get nested vm */
> +	ASSERT(hvm->nh_vmcx == NULL);
> +	hvm->nh_vmcx = hvm_map_guest_frame_ro(vmaddr >> PAGE_SHIFT);
> +	if (hvm->nh_vmcx == NULL) {
> +		gdprintk(XENLOG_ERR,
> +			"hvm_map_guest_frame_ro failed, injecting #GP\n");
> +		hvm_inject_exception(TRAP_gp_fault,
> +			HVM_DELIVER_NO_ERROR_CODE, 0);
> +		hvm->nh_hostflags.fields.vmentry = 0;
> +		return TRAP_gp_fault;
> +	}
> +
> +	/* save host state */
> +	ret = nhvm_vcpu_vmentry(v, regs, inst_len);
> +	if (ret) {
> +		gdprintk(XENLOG_ERR,
> +			"nhvm_vcpu_vmentry failed, injecting #UD\n");
> +		hvm_inject_exception(TRAP_invalid_op,
> +			HVM_DELIVER_NO_ERROR_CODE, 0);
> +		hvm->nh_hostflags.fields.vmentry = 0;
> +		hvm_unmap_guest_frame(hvm->nh_vmcx);
> +		hvm->nh_vmcx = NULL;
> +		return ret;
> +	}
> +
> +	hvm_unmap_guest_frame(hvm->nh_vmcx);
> +	hvm->nh_vmcx = NULL;
> +
> +	/* Switch vcpu to guest mode.
> +	 */
> +	nestedhvm_vcpu_enter_guestmode(v);
> +
> +	hvm->nh_hostflags.fields.vmentry = 0;
> +	return 0;
> +}
>
> You must not read VMX code yet or didn''t understand the reason why
VMX
> can;t do this. Appearantly this common code doesn''t apply to VMX
and have
> to be moved to arch specific stuff. VMX can only switch the context (L1 or
> L2 guest) at certain point (now defered to last step before resume) because
> only one VMCS can be accessed in one context.
>
> Thx, Eddie


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi!

This patch series brings Nested Virtualization to Xen.
This is the sixth patch series. Improvements to the
previous patch submission:

- Move GIF definition into SVM
- Move VMEXIT emulation into SVM
- Introduce hooks for getting host/guest cr3 for use with hap-on-hap
   per proposal from Eddie Dong
- Moved fields from struct nestedhvm into SVM
- Renamed struct nestedhvm to struct nestedvcpu
- Reworked VMRUN and VMEXIT emulation. It uses a defered emulation
   mechanism that makes interrupt handling more efficient and is closer
   to what VMX is doing
- VMCB is peristent mapped. Only remap the VMCB when l1 guest
   changes the address.


The patch series:

patch 01: add nestedhvm guest config option to the tools.
                  This is the only one patch touching the tools
patch 02: Add data structures for nested virtualization.
patch 03: add nestedhvm function hooks.
patch 04: The heart of nested virtualization.
patch 05: Allow switch to paged real mode during vmrun emulation.
                  Emulate cr0 and cr4 when guest does not intercept them
                  (i.e. Hyper-V/Windows7, KVM)
patch 06: When injecting an exception into nested guest, inject
                  #VMEXIT into the guest if intercepted.
patch 07: Allow guest to enable SVM in EFER only on AMD.
patch 08: Handle interrupts (generic part).
patch 09: SVM specific implementation for nested virtualization.
patch 10: Handle interrupts (SVM specific).
patch 11: The piece of code that effectively turns on nested virtualization.
patch 12: Move dirty_vram from struct hvm_domain to struct p2m_domain.
                  This change is the first part from a larger not-yet-ready
                  change where the vram and log_dirty tracking is teached
                  to work on per p2m.
patch 13: Handle nested pagefault to enable hap-on-hap and handle
                  nested guest page-table-walks to emulate instructions
                  the guest does not intercept (i.e. WBINVD with Windows 7).


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Christoph,

Thanks for these; I''ve commented on the individual patches, but mostly
the core nested hvm and nested svm looks like it''s in good shape.  The
nested HAP code still needs work based on the last review, though. 

Ccing Eddie for his opinion on the common code (patches 1 to 8).

Cheers,

Tim.

At 18:39 +0000 on 12 Nov (1289587157), Christoph Egger
wrote:
> Hi!
> 
> This patch series brings Nested Virtualization to Xen.
> This is the sixth patch series. Improvements to the
> previous patch submission:
> 
> - Move GIF definition into SVM
> - Move VMEXIT emulation into SVM
> - Introduce hooks for getting host/guest cr3 for use with hap-on-hap
>    per proposal from Eddie Dong
> - Moved fields from struct nestedhvm into SVM
> - Renamed struct nestedhvm to struct nestedvcpu
> - Reworked VMRUN and VMEXIT emulation. It uses a defered emulation
>    mechanism that makes interrupt handling more efficient and is closer
>    to what VMX is doing
> - VMCB is peristent mapped. Only remap the VMCB when l1 guest
>    changes the address.
> 
> 
> The patch series:
> 
> patch 01: add nestedhvm guest config option to the tools.
>                   This is the only one patch touching the tools
> patch 02: Add data structures for nested virtualization.
> patch 03: add nestedhvm function hooks.
> patch 04: The heart of nested virtualization.
> patch 05: Allow switch to paged real mode during vmrun emulation.
>                   Emulate cr0 and cr4 when guest does not intercept them
>                   (i.e. Hyper-V/Windows7, KVM)
> patch 06: When injecting an exception into nested guest, inject
>                   #VMEXIT into the guest if intercepted.
> patch 07: Allow guest to enable SVM in EFER only on AMD.
> patch 08: Handle interrupts (generic part).
> patch 09: SVM specific implementation for nested virtualization.
> patch 10: Handle interrupts (SVM specific).
> patch 11: The piece of code that effectively turns on nested
virtualization.
> patch 12: Move dirty_vram from struct hvm_domain to struct p2m_domain.
>                   This change is the first part from a larger not-yet-ready
>                   change where the vram and log_dirty tracking is teached
>                   to work on per p2m.
> patch 13: Handle nested pagefault to enable hap-on-hap and handle
>                   nested guest page-table-walks to emulate instructions
>                   the guest does not intercept (i.e. WBINVD with Windows
7).
> 
> 
> -- 
> ---to satisfy European Law for business letters:
> Advanced Micro Devices GmbH
> Einsteinring 24, 85609 Dornach b. Muenchen
> Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
> Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
> Registergericht Muenchen, HRB Nr. 43632
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Xen Platform Team
Citrix Systems UK Ltd.  (Company #02937203, SL9 0BG)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Christoph Egger wrote:
> Hi!
> 
> This patch series brings Nested Virtualization to Xen.
> This is the sixth patch series. Improvements to the
> previous patch submission:
> 
> - Move GIF definition into SVM
> - Move VMEXIT emulation into SVM
> - Introduce hooks for getting host/guest cr3 for use with hap-on-hap
>    per proposal from Eddie Dong
> - Moved fields from struct nestedhvm into SVM
> - Renamed struct nestedhvm to struct nestedvcpu
> - Reworked VMRUN and VMEXIT emulation. It uses a defered emulation
>    mechanism that makes interrupt handling more efficient and is
>    closer to what VMX is doing
> - VMCB is peristent mapped. Only remap the VMCB when l1 guest
>    changes the address.
> 
> 
> The patch series:
> 
> patch 01: add nestedhvm guest config option to the tools.
>                   This is the only one patch touching the tools
> patch 02: Add data structures for nested virtualization.
> patch 03: add nestedhvm function hooks.
> patch 04: The heart of nested virtualization.
> patch 05: Allow switch to paged real mode during vmrun emulation.
>                   Emulate cr0 and cr4 when guest does not intercept
>                   them (i.e. Hyper-V/Windows7, KVM)
> patch 06: When injecting an exception into nested guest, inject
>                   #VMEXIT into the guest if intercepted.
> patch 07: Allow guest to enable SVM in EFER only on AMD.
> patch 08: Handle interrupts (generic part).
> patch 09: SVM specific implementation for nested virtualization.
> patch 10: Handle interrupts (SVM specific).
> patch 11: The piece of code that effectively turns on nested
> virtualization. patch 12: Move dirty_vram from struct hvm_domain to
>                   struct p2m_domain. This change is the first part
>                   from a larger not-yet-ready change where the vram
>                   and log_dirty tracking is teached to work on per
> p2m. 
> patch 13: Handle nested pagefault to enable hap-on-hap and handle
>                   nested guest page-table-walks to emulate
>                   instructions the guest does not intercept (i.e.
> WBINVD with Windows 7). 
Thanks for the progress! 
I am fine with the patch series in general .

Some minor comments here:

+enum nestedhvm_intercepts {
+	/* exitinfo1 and exitinfo2 undefined */
+	NESTEDHVM_INTERCEPT_INVALID = 0, /* INVALID vmcb/vmcs */
+	NESTEDHVM_INTERCEPT_SHUTDOWN = 1, /* kill guest */
+	NESTEDHVM_INTERCEPT_MCE = 2, /* machine check exception */
+	NESTEDHVM_INTERCEPT_VMMCALL = 3, /* VMMCALL/VMCALL */
+
+	/* exitinfo1 is hvm_intsrc_*, exitinfo2 is the vector */
+	NESTEDHVM_INTERCEPT_INTR = 4, /* interrupt exit code */
+	NESTEDHVM_INTERCEPT_NMI = 5, /* NMI exit code */
+
+	/* exitinfo1 is PF error code, exitinfo2 is PF fault address */
+	NESTEDHVM_INTERCEPT_NPF = 6, /* nested page fault */
+	NESTEDHVM_INTERCEPT_PF = 7, /* page fault */
+
+	/* exceptions: exitinfo1 and exitinfo2 are undefined */
+	NESTEDHVM_INTERCEPT_NM = 8, /* device-not-available */
+
+	/* end mark */
+	NESTEDHVM_INTERCEPT_LAST,
+};

I didn''t see where it is used. Did I miss something?


+    union {
+        uint32_t bytes;
+        struct {
+            uint32_t vmentry_pending: 1;
+            uint32_t vmexit_pending: 1;
+            uint32_t vmswitch_in_progress: 1; /* true during vmentry/vmexit
emulation */
+            uint32_t reserved : 29;
+        } fields;
+    } nv_hostflags;

I feel it is unnecessary to use tricky bit definition (slow and readers may
think it has some relationship w/ hardware field). Using a simple byte variable
for each field seems more simple to me.


Both nhvm & nestedhvm is used in API names, what is the policy?

Thx, Eddie


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, Nov 12, 2010 at 10:39 AM, Christoph Egger
<Christoph.Egger@amd.com>wrote:
>
> Hi!
>
> This patch series brings Nested Virtualization to Xen.
> This is the sixth patch series. Improvements to the
> previous patch submission:
>
> - Move GIF definition into SVM
> - Move VMEXIT emulation into SVM
> - Introduce hooks for getting host/guest cr3 for use with hap-on-hap
>   per proposal from Eddie Dong
> - Moved fields from struct nestedhvm into SVM
> - Renamed struct nestedhvm to struct nestedvcpu
> - Reworked VMRUN and VMEXIT emulation. It uses a defered emulation
>   mechanism that makes interrupt handling more efficient and is closer
>   to what VMX is doing
> - VMCB is peristent mapped. Only remap the VMCB when l1 guest
>   changes the address.
>
>
> The patch series:
>
> patch 01: add nestedhvm guest config option to the tools.
>                  This is the only one patch touching the tools
> patch 02: Add data structures for nested virtualization.
> patch 03: add nestedhvm function hooks.
> patch 04: The heart of nested virtualization.
> patch 05: Allow switch to paged real mode during vmrun emulation.
>                  Emulate cr0 and cr4 when guest does not intercept them
>                  (i.e. Hyper-V/Windows7, KVM)
> patch 06: When injecting an exception into nested guest, inject
>                  #VMEXIT into the guest if intercepted.
> patch 07: Allow guest to enable SVM in EFER only on AMD.
> patch 08: Handle interrupts (generic part).
> patch 09: SVM specific implementation for nested virtualization.
> patch 10: Handle interrupts (SVM specific).
> patch 11: The piece of code that effectively turns on nested
> virtualization.
> patch 12: Move dirty_vram from struct hvm_domain to struct p2m_domain.
>                  This change is the first part from a larger not-yet-ready
>                  change where the vram and log_dirty tracking is teached
>                  to work on per p2m.
> patch 13: Handle nested pagefault to enable hap-on-hap and handle
>                  nested guest page-table-walks to emulate instructions
>                  the guest does not intercept (i.e. WBINVD with Windows 7).
>
>
> --
> ---to satisfy European Law for business letters:
> Advanced Micro Devices GmbH
> Einsteinring 24, 85609 Dornach b. Muenchen
> Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
> Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
> Registergericht Muenchen, HRB Nr. 43632
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

I''d like to ask a general question on nested virtualization. Can an L1
or
L>1 VM span multiple physical machines?
I''m guessing not from the pdf,
http://lists.xensource.com/archives/html/xen-devel/2010-04/pdfsVHwQIeZLB.pdf,
but I wanted to ask and confirm because if so, this would be a fantastic way
to segregate individual customer infrastructures in a cloud environment.

Thanks

-Bruce


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tuesday 30 November 2010 18:20:40 Bruce Edge wrote:
> On Fri, Nov 12, 2010 at 10:39 AM, Christoph Egger
> <Christoph.Egger@amd.com<mailto:Christoph.Egger@amd.com>>
wrote:
>
> Hi!
>
> This patch series brings Nested Virtualization to Xen.
> This is the sixth patch series. Improvements to the
> previous patch submission:
>
> - Move GIF definition into SVM
> - Move VMEXIT emulation into SVM
> - Introduce hooks for getting host/guest cr3 for use with hap-on-hap
>   per proposal from Eddie Dong
> - Moved fields from struct nestedhvm into SVM
> - Renamed struct nestedhvm to struct nestedvcpu
> - Reworked VMRUN and VMEXIT emulation. It uses a defered emulation
>   mechanism that makes interrupt handling more efficient and is closer
>   to what VMX is doing
> - VMCB is peristent mapped. Only remap the VMCB when l1 guest
>   changes the address.
>
>
> The patch series:
>
> patch 01: add nestedhvm guest config option to the tools.
>                  This is the only one patch touching the tools
> patch 02: Add data structures for nested virtualization.
> patch 03: add nestedhvm function hooks.
> patch 04: The heart of nested virtualization.
> patch 05: Allow switch to paged real mode during vmrun emulation.
>                  Emulate cr0 and cr4 when guest does not intercept them
>                  (i.e. Hyper-V/Windows7, KVM)
> patch 06: When injecting an exception into nested guest, inject
>                  #VMEXIT into the guest if intercepted.
> patch 07: Allow guest to enable SVM in EFER only on AMD.
> patch 08: Handle interrupts (generic part).
> patch 09: SVM specific implementation for nested virtualization.
> patch 10: Handle interrupts (SVM specific).
> patch 11: The piece of code that effectively turns on nested
> virtualization. patch 12: Move dirty_vram from struct hvm_domain to struct
> p2m_domain. This change is the first part from a larger not-yet-ready
> change where the vram and log_dirty tracking is teached to work on per p2m.
> patch 13: Handle nested pagefault to enable hap-on-hap and handle
>                  nested guest page-table-walks to emulate instructions
>                  the guest does not intercept (i.e. WBINVD with Windows 7).
>
>
> --
> ---to satisfy European Law for business letters:
> Advanced Micro Devices GmbH
> Einsteinring 24, 85609 Dornach b. Muenchen
> Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
> Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
> Registergericht Muenchen, HRB Nr. 43632
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com<mailto:Xen-devel@lists.xensource.com>
> http://lists.xensource.com/xen-devel
>
>
> I''d like to ask a general question on nested virtualization. Can
an L1 or
> L>1 VM span multiple physical machines?
I guess you mean virtual machines here. The answer is yes.
> I''m guessing not from the pdf, 
> http://lists.xensource.com/archives/html/xen-devel/2010-04/pdfsVHwQIeZLB.pd
>f, but I wanted to ask and confirm because if so, this would be a fantastic
> way to segregate individual customer infrastructures in a cloud
> environment.
I do not consider the patch series production stable but I would be glad when 
you can help with testing.

Christoph


-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Following up on Bruce''s question..
 What do you mean by VM spanning multiple physical machines?
 do you mean the ability to migrate VMs across hypervisor levels, across one
or more machines?

shriram
On Wed, Dec 1, 2010 at 1:08 AM, Christoph Egger
<Christoph.Egger@amd.com>wrote:
> On Tuesday 30 November 2010 18:20:40 Bruce Edge wrote:
> > On Fri, Nov 12, 2010 at 10:39 AM, Christoph Egger
> > <Christoph.Egger@amd.com<mailto:Christoph.Egger@amd.com>>
wrote:
> >
> > Hi!
> >
> > This patch series brings Nested Virtualization to Xen.
> > This is the sixth patch series. Improvements to the
> > previous patch submission:
> >
> > - Move GIF definition into SVM
> > - Move VMEXIT emulation into SVM
> > - Introduce hooks for getting host/guest cr3 for use with hap-on-hap
> >   per proposal from Eddie Dong
> > - Moved fields from struct nestedhvm into SVM
> > - Renamed struct nestedhvm to struct nestedvcpu
> > - Reworked VMRUN and VMEXIT emulation. It uses a defered emulation
> >   mechanism that makes interrupt handling more efficient and is closer
> >   to what VMX is doing
> > - VMCB is peristent mapped. Only remap the VMCB when l1 guest
> >   changes the address.
> >
> >
> > The patch series:
> >
> > patch 01: add nestedhvm guest config option to the tools.
> >                  This is the only one patch touching the tools
> > patch 02: Add data structures for nested virtualization.
> > patch 03: add nestedhvm function hooks.
> > patch 04: The heart of nested virtualization.
> > patch 05: Allow switch to paged real mode during vmrun emulation.
> >                  Emulate cr0 and cr4 when guest does not intercept
them
> >                  (i.e. Hyper-V/Windows7, KVM)
> > patch 06: When injecting an exception into nested guest, inject
> >                  #VMEXIT into the guest if intercepted.
> > patch 07: Allow guest to enable SVM in EFER only on AMD.
> > patch 08: Handle interrupts (generic part).
> > patch 09: SVM specific implementation for nested virtualization.
> > patch 10: Handle interrupts (SVM specific).
> > patch 11: The piece of code that effectively turns on nested
> > virtualization. patch 12: Move dirty_vram from struct hvm_domain to
> struct
> > p2m_domain. This change is the first part from a larger not-yet-ready
> > change where the vram and log_dirty tracking is teached to work on per
> p2m.
> > patch 13: Handle nested pagefault to enable hap-on-hap and handle
> >                  nested guest page-table-walks to emulate instructions
> >                  the guest does not intercept (i.e. WBINVD with
Windows
> 7).
> >
> >
> > --
> > ---to satisfy European Law for business letters:
> > Advanced Micro Devices GmbH
> > Einsteinring 24, 85609 Dornach b. Muenchen
> > Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
> > Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
> > Registergericht Muenchen, HRB Nr. 43632
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> >
Xen-devel@lists.xensource.com<mailto:Xen-devel@lists.xensource.com>
> > http://lists.xensource.com/xen-devel
> >
> >
> > I''d like to ask a general question on nested virtualization.
Can an L1 or
> > L>1 VM span multiple physical machines?
>
> I guess you mean virtual machines here. The answer is yes.
>
> > I''m guessing not from the pdf,
> >
> http://lists.xensource.com/archives/html/xen-devel/2010-04/pdfsVHwQIeZLB.pd
> >f, but I wanted to ask and confirm because if so, this would be a
> fantastic
> > way to segregate individual customer infrastructures in a cloud
> > environment.
>
> I do not consider the patch series production stable but I would be glad
> when
> you can help with testing.
>
> Christoph
>
>
> --
> ---to satisfy European Law for business letters:
> Advanced Micro Devices GmbH
> Einsteinring 24, 85609 Dornach b. Muenchen
> Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
> Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
> Registergericht Muenchen, HRB Nr. 43632
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>


-- 
perception is but an offspring of its own self


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Monday 22 November 2010 07:55:04 Dong, Eddie wrote:
> Christoph Egger wrote:
> > Hi!
> >
> > This patch series brings Nested Virtualization to Xen.
> > This is the sixth patch series. Improvements to the
> > previous patch submission:
> >
> > - Move GIF definition into SVM
> > - Move VMEXIT emulation into SVM
> > - Introduce hooks for getting host/guest cr3 for use with hap-on-hap
> >    per proposal from Eddie Dong
> > - Moved fields from struct nestedhvm into SVM
> > - Renamed struct nestedhvm to struct nestedvcpu
> > - Reworked VMRUN and VMEXIT emulation. It uses a defered emulation
> >    mechanism that makes interrupt handling more efficient and is
> >    closer to what VMX is doing
> > - VMCB is peristent mapped. Only remap the VMCB when l1 guest
> >    changes the address.
> >
> >
> > The patch series:
> >
> > patch 01: add nestedhvm guest config option to the tools.
> >                   This is the only one patch touching the tools
> > patch 02: Add data structures for nested virtualization.
> > patch 03: add nestedhvm function hooks.
> > patch 04: The heart of nested virtualization.
> > patch 05: Allow switch to paged real mode during vmrun emulation.
> >                   Emulate cr0 and cr4 when guest does not intercept
> >                   them (i.e. Hyper-V/Windows7, KVM)
> > patch 06: When injecting an exception into nested guest, inject
> >                   #VMEXIT into the guest if intercepted.
> > patch 07: Allow guest to enable SVM in EFER only on AMD.
> > patch 08: Handle interrupts (generic part).
> > patch 09: SVM specific implementation for nested virtualization.
> > patch 10: Handle interrupts (SVM specific).
> > patch 11: The piece of code that effectively turns on nested
> > virtualization. patch 12: Move dirty_vram from struct hvm_domain to
> >                   struct p2m_domain. This change is the first part
> >                   from a larger not-yet-ready change where the vram
> >                   and log_dirty tracking is teached to work on per
> > p2m.
> > patch 13: Handle nested pagefault to enable hap-on-hap and handle
> >                   nested guest page-table-walks to emulate
> >                   instructions the guest does not intercept (i.e.
> > WBINVD with Windows 7).
>
> Thanks for the progress!
> I am fine with the patch series in general .
>
> Some minor comments here:
>
> +enum nestedhvm_intercepts {
> +	/* exitinfo1 and exitinfo2 undefined */
> +	NESTEDHVM_INTERCEPT_INVALID = 0, /* INVALID vmcb/vmcs */
> +	NESTEDHVM_INTERCEPT_SHUTDOWN = 1, /* kill guest */
> +	NESTEDHVM_INTERCEPT_MCE = 2, /* machine check exception */
> +	NESTEDHVM_INTERCEPT_VMMCALL = 3, /* VMMCALL/VMCALL */
> +
> +	/* exitinfo1 is hvm_intsrc_*, exitinfo2 is the vector */
> +	NESTEDHVM_INTERCEPT_INTR = 4, /* interrupt exit code */
> +	NESTEDHVM_INTERCEPT_NMI = 5, /* NMI exit code */
> +
> +	/* exitinfo1 is PF error code, exitinfo2 is PF fault address */
> +	NESTEDHVM_INTERCEPT_NPF = 6, /* nested page fault */
> +	NESTEDHVM_INTERCEPT_PF = 7, /* page fault */
> +
> +	/* exceptions: exitinfo1 and exitinfo2 are undefined */
> +	NESTEDHVM_INTERCEPT_NM = 8, /* device-not-available */
> +
> +	/* end mark */
> +	NESTEDHVM_INTERCEPT_LAST,
> +};
>
> I didn''t see where it is used. Did I miss something?
I forgot to remove them. Thanks for pointing this out. Fixed.
>
>
> +    union {
> +        uint32_t bytes;
> +        struct {
> +            uint32_t vmentry_pending: 1;
> +            uint32_t vmexit_pending: 1;
> +            uint32_t vmswitch_in_progress: 1; /* true during
> vmentry/vmexit emulation */ +            uint32_t reserved : 29;
> +        } fields;
> +    } nv_hostflags;
>
> I feel it is unnecessary to use tricky bit definition (slow and readers may
> think it has some relationship w/ hardware field). Using a simple byte
> variable for each field seems more simple to me.
hmm... hw bit field in common code?
Anyway, I don''t mind. I will change it as you suggest.
>
> Both nhvm & nestedhvm is used in API names, what is the policy?
nhvm just call the function hooks.
Likewise within SVM: nsvm implements the interface.

Christoph



-- 
---to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Einsteinring 24, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel