Hi All,
This the a nested virtualization test report for Xen 4.3-RC1 on Intel hardware.
We use Linux 3.9.1 as Dom0.
a. Virtual EPT and VMCS shadowing features can work fine.
b. Xen, KVM and VMware can basically work on top of L0 Xen.
c. 32bit/64bit Linux and Windows are covered as L2 guests.
There are three basic entities in Xen nested virtualization.
L0: Xen (64bit Xen and 64bit Dom0), which is at the bottom of the nested stack.
L1: Xen or KVM or VMware or VirtualBox (all in 64bit mode)
L2: Linux or Windows guest, which is at the top of the nested stack.
(when saying ''KVM on Xen'', I mean L0 hypervisor is Xen and L1
hypervisor is KVM.)
Workable cases: (Pass)
1. virtual EPT and VMCS shadowing feature enabled
2. 64bit Linux/Windows as L2 guest for "Xen on Xen"
3. 64bit Linux guest as L2 guest for "KVM on Xen"
4. L1 KVM and L1 Xen simultaneously running on a L0 Xen
5. L2 guest Save/Restore and local migration for "KVM on Xen"
6. AVX and XSAVE in L2 guest for "KVM on Xen"
7. some workloads (e.g. LTP, Kernel-build, UnixBench) can work fine in 64bit L2
Linux guest
8. 64bit Linux L2 guest can boot up for "VMware on Xen"
9. 32bit L2 guest (Linux/Windows) booting on "Xen on Xen" (not use
EPT in L1)
10. 32bit/64bit Windows and 32bit Linux L2 guest booting on "VMware on
Xen" (not use EPT in L1)
N.B. Only if you don''t use EPT feature in L1 hypervisor, case #9 and
#10 can work fine.
Non-workable cases: (Fail)
1. 32bit/64bit Windows L2 guest booting on "KVM on Xen"
2. L2 guest Save/Restore and local migration for "Xen on Xen"
3. Migration "from L0 to L1" for "Xen on Xen"
4. Migration "from L1 to L0" for "Xen on Xen"
5. Migration a L1 Xen/KVM guest with a L2 running in that L1
6. L2 guest booting on "VirtualBox on Xen"
Best Regards,
Yongjie (Jay)
On Fri, May 10, 2013 at 12:07 PM, Ren, Yongjie <yongjie.ren@intel.com> wrote:> Hi All, > This the a nested virtualization test report for Xen 4.3-RC1 on Intel hardware. We use Linux 3.9.1 as Dom0. > a. Virtual EPT and VMCS shadowing features can work fine. > b. Xen, KVM and VMware can basically work on top of L0 Xen. > c. 32bit/64bit Linux and Windows are covered as L2 guests.Sorry I just saw this -- thanks for the nice enumeration. Two questions. First, I don''t see the Win7 "XP compatibility mode" on this list -- that would be L0 Xen, L1 Win7, L2 XP. This seems like probably the most likely actual real-world use of nested virt. Is that on your radar at all? Secondly, what do you think is the primary use case for Xen-on-Xen (or KVM-on-Xen, &c)? Who would want to use it and why? Thanks, -George> > There are three basic entities in Xen nested virtualization. > L0: Xen (64bit Xen and 64bit Dom0), which is at the bottom of the nested stack. > L1: Xen or KVM or VMware or VirtualBox (all in 64bit mode) > L2: Linux or Windows guest, which is at the top of the nested stack. > (when saying ''KVM on Xen'', I mean L0 hypervisor is Xen and L1 hypervisor is KVM.) > > Workable cases: (Pass) > 1. virtual EPT and VMCS shadowing feature enabled > 2. 64bit Linux/Windows as L2 guest for "Xen on Xen" > 3. 64bit Linux guest as L2 guest for "KVM on Xen" > 4. L1 KVM and L1 Xen simultaneously running on a L0 Xen > 5. L2 guest Save/Restore and local migration for "KVM on Xen" > 6. AVX and XSAVE in L2 guest for "KVM on Xen" > 7. some workloads (e.g. LTP, Kernel-build, UnixBench) can work fine in 64bit L2 Linux guest > 8. 64bit Linux L2 guest can boot up for "VMware on Xen" > 9. 32bit L2 guest (Linux/Windows) booting on "Xen on Xen" (not use EPT in L1) > 10. 32bit/64bit Windows and 32bit Linux L2 guest booting on "VMware on Xen" (not use EPT in L1) > N.B. Only if you don''t use EPT feature in L1 hypervisor, case #9 and #10 can work fine. > > Non-workable cases: (Fail) > 1. 32bit/64bit Windows L2 guest booting on "KVM on Xen" > 2. L2 guest Save/Restore and local migration for "Xen on Xen" > 3. Migration "from L0 to L1" for "Xen on Xen" > 4. Migration "from L1 to L0" for "Xen on Xen" > 5. Migration a L1 Xen/KVM guest with a L2 running in that L1 > 6. L2 guest booting on "VirtualBox on Xen" > > > Best Regards, > Yongjie (Jay) > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Pasi Kärkkäinen
2013-Jun-27 12:46 UTC
Re: nested virtualizaiton test report for Xen 4.3-RC1
On Thu, Jun 27, 2013 at 12:37:56PM +0100, George Dunlap wrote:> On Fri, May 10, 2013 at 12:07 PM, Ren, Yongjie <yongjie.ren@intel.com> wrote: > > Hi All, > > This the a nested virtualization test report for Xen 4.3-RC1 on Intel hardware. We use Linux 3.9.1 as Dom0. > > a. Virtual EPT and VMCS shadowing features can work fine. > > b. Xen, KVM and VMware can basically work on top of L0 Xen. > > c. 32bit/64bit Linux and Windows are covered as L2 guests. > > Sorry I just saw this -- thanks for the nice enumeration. > > Two questions. First, I don''t see the Win7 "XP compatibility mode" on > this list -- that would be L0 Xen, L1 Win7, L2 XP. This seems like > probably the most likely actual real-world use of nested virt. Is > that on your radar at all? > > Secondly, what do you think is the primary use case for Xen-on-Xen (or > KVM-on-Xen, &c)? Who would want to use it and why? >I can think of at least two use-cases: - Xen-on-Xen might be good for testing/debugging the hypervisor.. Much easier to crash and debug the virtual Xen rather than physical machine. - Xen-on-Xen makes it possible to create easy-and-fast-to-clone lab/poc/test environments with "full" functionality thanks to virtual vmx and ept.. -- Pasi> Thanks, > -George > > > > > There are three basic entities in Xen nested virtualization. > > L0: Xen (64bit Xen and 64bit Dom0), which is at the bottom of the nested stack. > > L1: Xen or KVM or VMware or VirtualBox (all in 64bit mode) > > L2: Linux or Windows guest, which is at the top of the nested stack. > > (when saying ''KVM on Xen'', I mean L0 hypervisor is Xen and L1 hypervisor is KVM.) > > > > Workable cases: (Pass) > > 1. virtual EPT and VMCS shadowing feature enabled > > 2. 64bit Linux/Windows as L2 guest for "Xen on Xen" > > 3. 64bit Linux guest as L2 guest for "KVM on Xen" > > 4. L1 KVM and L1 Xen simultaneously running on a L0 Xen > > 5. L2 guest Save/Restore and local migration for "KVM on Xen" > > 6. AVX and XSAVE in L2 guest for "KVM on Xen" > > 7. some workloads (e.g. LTP, Kernel-build, UnixBench) can work fine in 64bit L2 Linux guest > > 8. 64bit Linux L2 guest can boot up for "VMware on Xen" > > 9. 32bit L2 guest (Linux/Windows) booting on "Xen on Xen" (not use EPT in L1) > > 10. 32bit/64bit Windows and 32bit Linux L2 guest booting on "VMware on Xen" (not use EPT in L1) > > N.B. Only if you don''t use EPT feature in L1 hypervisor, case #9 and #10 can work fine. > > > > Non-workable cases: (Fail) > > 1. 32bit/64bit Windows L2 guest booting on "KVM on Xen" > > 2. L2 guest Save/Restore and local migration for "Xen on Xen" > > 3. Migration "from L0 to L1" for "Xen on Xen" > > 4. Migration "from L1 to L0" for "Xen on Xen" > > 5. Migration a L1 Xen/KVM guest with a L2 running in that L1 > > 6. L2 guest booting on "VirtualBox on Xen" > > > > > > Best Regards, > > Yongjie (Jay) > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
Aravindh Puthiyaparambil (aravindp)
2013-Jun-27 18:30 UTC
Re: nested virtualizaiton test report for Xen 4.3-RC1
> On Thu, Jun 27, 2013 at 12:37:56PM +0100, George Dunlap wrote: > > On Fri, May 10, 2013 at 12:07 PM, Ren, Yongjie <yongjie.ren@intel.com> > wrote: > > > Hi All, > > > This the a nested virtualization test report for Xen 4.3-RC1 on Intel > hardware. We use Linux 3.9.1 as Dom0. > > > a. Virtual EPT and VMCS shadowing features can work fine. > > > b. Xen, KVM and VMware can basically work on top of L0 Xen. > > > c. 32bit/64bit Linux and Windows are covered as L2 guests. > > > > Sorry I just saw this -- thanks for the nice enumeration. > > > > Two questions. First, I don''t see the Win7 "XP compatibility mode" on > > this list -- that would be L0 Xen, L1 Win7, L2 XP. This seems like > > probably the most likely actual real-world use of nested virt. Is > > that on your radar at all? > > > > Secondly, what do you think is the primary use case for Xen-on-Xen (or > > KVM-on-Xen, &c)? Who would want to use it and why? > > > > I can think of at least two use-cases: > > - Xen-on-Xen might be good for testing/debugging the hypervisor.. Much > easier to crash and debug the virtual Xen rather than physical machine.I find this very useful. I am doing all my Xen development and testing in a Xen-on-VMware environment because the ESX nested support seems to be fully baked. I am hoping nested support gets better so that I can move to a Xen-on-Xen environment. In fact once Xen 4.3 is released I will move to this setup so that I can actively test it and provide feedback / fixes. Thanks, Aravindh> - Xen-on-Xen makes it possible to create easy-and-fast-to-clone lab/poc/test > environments with "full" functionality thanks to virtual vmx and ept.. > > > -- Pasi > > > > Thanks, > > -George > > > > > > > > There are three basic entities in Xen nested virtualization. > > > L0: Xen (64bit Xen and 64bit Dom0), which is at the bottom of the > nested stack. > > > L1: Xen or KVM or VMware or VirtualBox (all in 64bit mode) > > > L2: Linux or Windows guest, which is at the top of the nested stack. > > > (when saying ''KVM on Xen'', I mean L0 hypervisor is Xen and L1 > > > hypervisor is KVM.) > > > > > > Workable cases: (Pass) > > > 1. virtual EPT and VMCS shadowing feature enabled 2. 64bit > > > Linux/Windows as L2 guest for "Xen on Xen" > > > 3. 64bit Linux guest as L2 guest for "KVM on Xen" > > > 4. L1 KVM and L1 Xen simultaneously running on a L0 Xen 5. L2 guest > > > Save/Restore and local migration for "KVM on Xen" > > > 6. AVX and XSAVE in L2 guest for "KVM on Xen" > > > 7. some workloads (e.g. LTP, Kernel-build, UnixBench) can work fine > > > in 64bit L2 Linux guest 8. 64bit Linux L2 guest can boot up for "VMware on > Xen" > > > 9. 32bit L2 guest (Linux/Windows) booting on "Xen on Xen" (not use > > > EPT in L1) 10. 32bit/64bit Windows and 32bit Linux L2 guest booting > > > on "VMware on Xen" (not use EPT in L1) N.B. Only if you don''t use EPT > feature in L1 hypervisor, case #9 and #10 can work fine. > > > > > > Non-workable cases: (Fail) > > > 1. 32bit/64bit Windows L2 guest booting on "KVM on Xen" > > > 2. L2 guest Save/Restore and local migration for "Xen on Xen" > > > 3. Migration "from L0 to L1" for "Xen on Xen" > > > 4. Migration "from L1 to L0" for "Xen on Xen" > > > 5. Migration a L1 Xen/KVM guest with a L2 running in that L1 6. L2 > > > guest booting on "VirtualBox on Xen" > > > > > > > > > Best Regards, > > > Yongjie (Jay) > > > > > > > > > > > > _______________________________________________ > > > Xen-devel mailing list > > > Xen-devel@lists.xen.org > > > http://lists.xen.org/xen-devel > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
On Thu, Jun 27, 2013 at 12:37:56PM +0100, George Dunlap wrote:> On Fri, May 10, 2013 at 12:07 PM, Ren, Yongjie <yongjie.ren@intel.com> wrote: > > Hi All, > > This the a nested virtualization test report for Xen 4.3-RC1 on Intel hardware. We use Linux 3.9.1 as Dom0. > > a. Virtual EPT and VMCS shadowing features can work fine. > > b. Xen, KVM and VMware can basically work on top of L0 Xen. > > c. 32bit/64bit Linux and Windows are covered as L2 guests. > > Sorry I just saw this -- thanks for the nice enumeration. > > Two questions. First, I don''t see the Win7 "XP compatibility mode" on > this list -- that would be L0 Xen, L1 Win7, L2 XP. This seems like > probably the most likely actual real-world use of nested virt. Is > that on your radar at all? > > Secondly, what do you think is the primary use case for Xen-on-Xen (or > KVM-on-Xen, &c)? Who would want to use it and why?One use case is u-Xen (used by Bromium) on XenClient XT. Who could use it: XC-XT users who isolate VM workloads of different security levels, who want to isolate specific tasks (e.g. web browsing) within a single VM. Why would they use it? For defense in depth, XC-XT could provide VM isolation (boot-time TXT measured launch and VT-d isolation of NICs) while u-Xen could provide run-time task separation within an isolated VM. Rich
On Thu, Jun 27, 2013 at 12:37:56PM +0100, George Dunlap wrote:> On Fri, May 10, 2013 at 12:07 PM, Ren, Yongjie <yongjie.ren@intel.com> wrote: > > Hi All, > > This the a nested virtualization test report for Xen 4.3-RC1 on Intel hardware. We use Linux 3.9.1 as Dom0. > > a. Virtual EPT and VMCS shadowing features can work fine. > > b. Xen, KVM and VMware can basically work on top of L0 Xen. > > c. 32bit/64bit Linux and Windows are covered as L2 guests. > > Sorry I just saw this -- thanks for the nice enumeration. > > Two questions. First, I don''t see the Win7 "XP compatibility mode" on > this list -- that would be L0 Xen, L1 Win7, L2 XP. This seems like > probably the most likely actual real-world use of nested virt. Is > that on your radar at all? > > Secondly, what do you think is the primary use case for Xen-on-Xen (or > KVM-on-Xen, &c)? Who would want to use it and why?One use case is u-Xen (used by Bromium) on XenClient XT. Who could use it: XC-XT users who isolate VM workloads of different security levels, who want to isolate specific tasks (e.g. web browsing) within a single VM. Why would they use it? For defense in depth, XC-XT could provide VM isolation (boot-time TXT measured launch and VT-d isolation of NICs) while u-Xen could provide run-time task separation within an isolated VM. Rich