Jeroen Groenewegen van der Weyden
2011-Feb-14 17:23 UTC
[Xen-users] Nested Virtualisation question
Hello, Anybody here has any experience with nested virtual machines. In particular I''m looking to run KVM in a DOMU. Anybody any experience with this mfg, Jeroen _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, Feb 14, 2011 at 12:23 PM, Jeroen Groenewegen van der Weyden < groen692@grosc.com> wrote:> Hello, > > Anybody here has any experience with nested virtual machines. In particular > I''m looking to run KVM in a DOMU. Anybody any experience with this >Ok first question is Why ?> mfg, > Jeroen > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On 02/14/2011 12:23 PM, Jeroen Groenewegen van der Weyden wrote:> Hello, > > Anybody here has any experience with nested virtual machines. In > particular I''m looking to run KVM in a DOMU. Anybody any experience with > this > > mfg, > JeroenI tried it once, out of curiosity. It didn''t work well, and I didn''t try very hard. I would guess that you would need to fully virtualize, and I expect performance would be terrible. -- Digimer E-Mail: digimer@alteeve.com AN!Whitepapers: http://alteeve.com Node Assassin: http://nodeassassin.org _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeroen Groenewegen van der Weyden
2011-Feb-15 06:47 UTC
Re: [Xen-users] Nested Virtualisation question
The why is a good question, We use xen as out main virtualisation platform. But now there are applications that use virtualisation it self. In my case I have an application that uses kvm. So to put it on our xen server it needs to be nested. I guess in the future more of these situation will appear. I read somethings about nested virtualisation and I wonder If somebody has any experience with it. mfg, jeroen Op 14-2-2011 18:58, Outback Dingo schreef:> On Mon, Feb 14, 2011 at 12:23 PM, Jeroen Groenewegen van der Weyden > <groen692@grosc.com <mailto:groen692@grosc.com>> wrote: > > Hello, > > Anybody here has any experience with nested virtual machines. In > particular I''m looking to run KVM in a DOMU. Anybody any > experience with this > > > Ok first question is Why ? > > mfg, > Jeroen > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com <mailto:Xen-users@lists.xensource.com> > http://lists.xensource.com/xen-users > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden <groen692@grosc.com> wrote:> The why is a good question, > We use xen as out main virtualisation platform. But now there are > applications that use virtualisation it self. In my case I have an > application that uses kvm. So to put it on our xen server it needs to be > nested. I guess in the future more of these situation will appear. > > I read somethings about nested virtualisation and I wonder If somebody has > any experience with it. >To run KVM inside of a guest you need virtualization support within the guest (running nested xen doesn''t require the double nested (unless you want to do HVM within the nested of course)) Patches can be found here: http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results I believe KVM should also have support for nested vmx and nested svm. The use cases are fairly slim, so finding users/testers is often difficult. Please report any bugs you find to the appropriate devel lists and cc the appropriate nested virtualization maintainers . Hope that helps. Thanks, Todd _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden <groen692@grosc.com> wrote:> I now am in the position to start testing your suggestion. > > An additional question, is it correct the patches you mention here are part > of the xen4.2-unstable branch? > are these patches for vt-x or for svm only? the developer seems to be from > amd. My machines are running on intel. >The latest patch series is here: http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview-td3415672.html Not sure if it has been accepted yet or not. You can email the developer and/or devel list for confirmation. Thanks, Todd> mfg, > Jeroen > > Op 15-2-2011 16:21, Todd Deshane schreef: >> >> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden >> <groen692@grosc.com> wrote: >>> >>> The why is a good question, >>> We use xen as out main virtualisation platform. But now there are >>> applications that use virtualisation it self. In my case I have an >>> application that uses kvm. So to put it on our xen server it needs to be >>> nested. I guess in the future more of these situation will appear. >>> >>> I read somethings about nested virtualisation and I wonder If somebody >>> has >>> any experience with it. >>> >> To run KVM inside of a guest you need virtualization support within >> the guest (running nested xen doesn''t require the double nested >> (unless you want to do HVM within the nested of course)) >> >> Patches can be found here: >> >> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results >> >> I believe KVM should also have support for nested vmx and nested svm. >> >> The use cases are fairly slim, so finding users/testers is often >> difficult. >> >> Please report any bugs you find to the appropriate devel lists and cc >> the appropriate nested virtualization maintainers . >> >> Hope that helps. >> >> Thanks, >> Todd >> >> _______________________________________________ >> Xen-users mailing list >> Xen-users@lists.xensource.com >> http://lists.xensource.com/xen-users >> > >-- Todd Deshane http://www.linkedin.com/in/deshantm http://www.xen.org/products/cloudxen.html _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Jeroen Groenewegen van der Weyden
2011-Apr-04 06:47 UTC
[Xen-devel] Re: [Xen-users] Nested Virtualisation question
Hello, I have A question about nested virtualisation, the xen user list suggested to post the question here. I need nested virtualisation to run an application on out en environment, which uses kvm it-self. I suppose I have to apply these patches: - http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results - or http://xenbits.xen.org/ext/cegger/nestedhvm/ Is it correct these patches are now in the xen-unstable branch? Anything else to get me on my way. My goal is DOM0 openSUSE 11.4 with xen 4.X DOMU1 SLES 11 SP1 kvm guest HVM domu1a (propetary OS) kvm guest HVM domu1b (sles 10 sp3) kvm guest HVM domu1c (sles 10 sp3) kvm guest HVM domu1b (sles 10 sp3) Op 3-4-2011 23:29, Todd Deshane schreef:> On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden > <groen692@grosc.com> wrote: >> I now am in the position to start testing your suggestion. >> >> An additional question, is it correct the patches you mention here are part >> of the xen4.2-unstable branch? >> are these patches for vt-x or for svm only? the developer seems to be from >> amd. My machines are running on intel. >> > The latest patch series is here: > http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview-td3415672.html > > Not sure if it has been accepted yet or not. You can email the > developer and/or devel list for confirmation. > > Thanks, > Todd > >> mfg, >> Jeroen >> >> Op 15-2-2011 16:21, Todd Deshane schreef: >>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden >>> <groen692@grosc.com> wrote: >>>> The why is a good question, >>>> We use xen as out main virtualisation platform. But now there are >>>> applications that use virtualisation it self. In my case I have an >>>> application that uses kvm. So to put it on our xen server it needs to be >>>> nested. I guess in the future more of these situation will appear. >>>> >>>> I read somethings about nested virtualisation and I wonder If somebody >>>> has >>>> any experience with it. >>>> >>> To run KVM inside of a guest you need virtualization support within >>> the guest (running nested xen doesn''t require the double nested >>> (unless you want to do HVM within the nested of course)) >>> >>> Patches can be found here: >>> >>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results >>> >>> I believe KVM should also have support for nested vmx and nested svm. >>> >>> The use cases are fairly slim, so finding users/testers is often >>> difficult. >>> >>> Please report any bugs you find to the appropriate devel lists and cc >>> the appropriate nested virtualization maintainers . >>> >>> Hope that helps. >>> >>> Thanks, >>> Todd >>> >>> _______________________________________________ >>> Xen-users mailing list >>> Xen-users@lists.xensource.com >>> http://lists.xensource.com/xen-users >>> >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeroen Groenewegen van der Weyden
2011-Apr-04 06:49 UTC
[Xen-devel] Nested Virtualisation question
Hello, I have A question about nested virtualisation, the xen user list suggested to post the question here. I need nested virtualisation to run an application on out en environment, which uses kvm it-self. I suppose I have to apply these patches: - http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results - or http://xenbits.xen.org/ext/cegger/nestedhvm/ Is it correct these patches are now in the xen-unstable branch? Anything else to get me on my way. My goal is DOM0 openSUSE 11.4 with xen 4.X DOMU1 SLES 11 SP1 kvm guest HVM domu1a (propetary OS) kvm guest HVM domu1b (sles 10 sp3) kvm guest HVM domu1c (sles 10 sp3) kvm guest HVM domu1b (sles 10 sp3)> Op 3-4-2011 23:29, Todd Deshane schreef: >> On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden >> <groen692@grosc.com> wrote: >>> I now am in the position to start testing your suggestion. >>> >>> An additional question, is it correct the patches you mention here >>> are part >>> of the xen4.2-unstable branch? >>> are these patches for vt-x or for svm only? the developer seems to >>> be from >>> amd. My machines are running on intel. >>> >> The latest patch series is here: >> http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview-td3415672.html >> >> >> Not sure if it has been accepted yet or not. You can email the >> developer and/or devel list for confirmation. >> >> Thanks, >> Todd >> >>> mfg, >>> Jeroen >>> >>> Op 15-2-2011 16:21, Todd Deshane schreef: >>>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden >>>> <groen692@grosc.com> wrote: >>>>> The why is a good question, >>>>> We use xen as out main virtualisation platform. But now there are >>>>> applications that use virtualisation it self. In my case I have an >>>>> application that uses kvm. So to put it on our xen server it needs >>>>> to be >>>>> nested. I guess in the future more of these situation will appear. >>>>> >>>>> I read somethings about nested virtualisation and I wonder If >>>>> somebody >>>>> has >>>>> any experience with it. >>>>> >>>> To run KVM inside of a guest you need virtualization support within >>>> the guest (running nested xen doesn''t require the double nested >>>> (unless you want to do HVM within the nested of course)) >>>> >>>> Patches can be found here: >>>> >>>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results >>>> >>>> >>>> I believe KVM should also have support for nested vmx and nested svm. >>>> >>>> The use cases are fairly slim, so finding users/testers is often >>>> difficult. >>>> >>>> Please report any bugs you find to the appropriate devel lists and cc >>>> the appropriate nested virtualization maintainers . >>>> >>>> Hope that helps. >>>> >>>> Thanks, >>>> Todd >>>> >>>> _______________________________________________ >>>> Xen-users mailing list >>>> Xen-users@lists.xensource.com >>>> http://lists.xensource.com/xen-users >>>> >>> >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2011-Apr-04 09:51 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
At 07:47 +0100 on 04 Apr (1301903251), Jeroen Groenewegen van der Weyden wrote:> Hello, > > I have A question about nested virtualisation, the xen user list > suggested to post the question here. > I need nested virtualisation to run an application on out en > environment, which uses kvm it-self. > > I suppose I have to apply these patches: > - > http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results > - or http://xenbits.xen.org/ext/cegger/nestedhvm/ > > Is it correct these patches are now in the xen-unstable branch?Not just yet. There were two problems outstanding with patch #12 of the latest version (a TLB flush race and a subsequent deadlock in the fix), which are both fixed. As soon as Christoph posts his fixes, the patch series will be applied. Cheers, Tim.> Anything else to get me on my way. > > > My goal is > > DOM0 > openSUSE 11.4 with xen 4.X > > DOMU1 > SLES 11 SP1 > kvm guest HVM domu1a (propetary OS) > kvm guest HVM domu1b (sles 10 sp3) > kvm guest HVM domu1c (sles 10 sp3) > kvm guest HVM domu1b (sles 10 sp3) > > Op 3-4-2011 23:29, Todd Deshane schreef: > > On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden > > <groen692@grosc.com> wrote: > >> I now am in the position to start testing your suggestion. > >> > >> An additional question, is it correct the patches you mention here are part > >> of the xen4.2-unstable branch? > >> are these patches for vt-x or for svm only? the developer seems to be from > >> amd. My machines are running on intel. > >> > > The latest patch series is here: > > http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview-td3415672.html > > > > Not sure if it has been accepted yet or not. You can email the > > developer and/or devel list for confirmation. > > > > Thanks, > > Todd > > > >> mfg, > >> Jeroen > >> > >> Op 15-2-2011 16:21, Todd Deshane schreef: > >>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden > >>> <groen692@grosc.com> wrote: > >>>> The why is a good question, > >>>> We use xen as out main virtualisation platform. But now there are > >>>> applications that use virtualisation it self. In my case I have an > >>>> application that uses kvm. So to put it on our xen server it needs to be > >>>> nested. I guess in the future more of these situation will appear. > >>>> > >>>> I read somethings about nested virtualisation and I wonder If somebody > >>>> has > >>>> any experience with it. > >>>> > >>> To run KVM inside of a guest you need virtualization support within > >>> the guest (running nested xen doesn''t require the double nested > >>> (unless you want to do HVM within the nested of course)) > >>> > >>> Patches can be found here: > >>> > >>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results > >>> > >>> I believe KVM should also have support for nested vmx and nested svm. > >>> > >>> The use cases are fairly slim, so finding users/testers is often > >>> difficult. > >>> > >>> Please report any bugs you find to the appropriate devel lists and cc > >>> the appropriate nested virtualization maintainers . > >>> > >>> Hope that helps. > >>> > >>> Thanks, > >>> Todd > >>> > >>> _______________________________________________ > >>> Xen-users mailing list > >>> Xen-users@lists.xensource.com > >>> http://lists.xensource.com/xen-users > >>> > >> > > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel-- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Xen Platform Team Citrix Systems UK Ltd. (Company #02937203, SL9 0BG) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeroen Groenewegen van der Weyden
2011-Apr-04 10:08 UTC
Re: [Xen-devel] Nested Virtualisation question
Ok thank you, Anything I can do, to test or something (I do not want to sound like I''m in a hurry) mfg, jeroen Op 4-4-2011 11:51, Tim Deegan schreef:> At 07:47 +0100 on 04 Apr (1301903251), Jeroen Groenewegen van der Weyden wrote: >> Hello, >> >> I have A question about nested virtualisation, the xen user list >> suggested to post the question here. >> I need nested virtualisation to run an application on out en >> environment, which uses kvm it-self. >> >> I suppose I have to apply these patches: >> - >> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results >> - or http://xenbits.xen.org/ext/cegger/nestedhvm/ >> >> Is it correct these patches are now in the xen-unstable branch? > Not just yet. There were two problems outstanding with patch #12 of the > latest version (a TLB flush race and a subsequent deadlock in the fix), > which are both fixed. As soon as Christoph posts his fixes, the patch > series will be applied. > > Cheers, > > Tim. > >> Anything else to get me on my way. >> >> >> My goal is >> >> DOM0 >> openSUSE 11.4 with xen 4.X >> >> DOMU1 >> SLES 11 SP1 >> kvm guest HVM domu1a (propetary OS) >> kvm guest HVM domu1b (sles 10 sp3) >> kvm guest HVM domu1c (sles 10 sp3) >> kvm guest HVM domu1b (sles 10 sp3) >> >> Op 3-4-2011 23:29, Todd Deshane schreef: >>> On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden >>> <groen692@grosc.com> wrote: >>>> I now am in the position to start testing your suggestion. >>>> >>>> An additional question, is it correct the patches you mention here are part >>>> of the xen4.2-unstable branch? >>>> are these patches for vt-x or for svm only? the developer seems to be from >>>> amd. My machines are running on intel. >>>> >>> The latest patch series is here: >>> http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview-td3415672.html >>> >>> Not sure if it has been accepted yet or not. You can email the >>> developer and/or devel list for confirmation. >>> >>> Thanks, >>> Todd >>> >>>> mfg, >>>> Jeroen >>>> >>>> Op 15-2-2011 16:21, Todd Deshane schreef: >>>>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden >>>>> <groen692@grosc.com> wrote: >>>>>> The why is a good question, >>>>>> We use xen as out main virtualisation platform. But now there are >>>>>> applications that use virtualisation it self. In my case I have an >>>>>> application that uses kvm. So to put it on our xen server it needs to be >>>>>> nested. I guess in the future more of these situation will appear. >>>>>> >>>>>> I read somethings about nested virtualisation and I wonder If somebody >>>>>> has >>>>>> any experience with it. >>>>>> >>>>> To run KVM inside of a guest you need virtualization support within >>>>> the guest (running nested xen doesn''t require the double nested >>>>> (unless you want to do HVM within the nested of course)) >>>>> >>>>> Patches can be found here: >>>>> >>>>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results >>>>> >>>>> I believe KVM should also have support for nested vmx and nested svm. >>>>> >>>>> The use cases are fairly slim, so finding users/testers is often >>>>> difficult. >>>>> >>>>> Please report any bugs you find to the appropriate devel lists and cc >>>>> the appropriate nested virtualization maintainers . >>>>> >>>>> Hope that helps. >>>>> >>>>> Thanks, >>>>> Todd >>>>> >>>>> _______________________________________________ >>>>> Xen-users mailing list >>>>> Xen-users@lists.xensource.com >>>>> http://lists.xensource.com/xen-users >>>>> >>> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeroen Groenewegen van der Weyden
2011-Apr-07 05:36 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
I couldn''t help noticing some patches were added in the /staging/xen-unstable branch. So I gave it a go. I know it is not called the unstable branch for nothing but, but I want to let you know that it the entire system crashes when any domain is created: xl create someDomain -> results in immediate system crash System: - domO system openSUSE 11.4 x86_64. kernel 2.6.37.1-1.2-xen - xen build from staging/xen-unstable (c/s 23172) - proc intel q9450 +++++++++++++console output hydra:/etc/xen/vm # xl create groscWindowsXpWorkstation Parsing config file groscWindowsXpWorkstation xc: info: VIRTUAL MEMORY ARRANGEMENT: Loader: 0000000000100000->00000000001795b0 TOTAL: 0000000000000000->0000000017800000 ENTRY ADDRESS: 0000000000100000 xc: info: PHYSICAL MEMORY ALLOCATION: 4KB PAGES: 0x0000000000000200 2MB PAGES: 0x00000000000000bb 1GB PAGES: 0x0000000000000000 libxl: error: libxl_device.c:475:libxl__wait_for_device_model Device Model not ready xl: fatal error: libxl_create.c:532, rc=-1: libxl__confirm_device_model_startup +++++++++++++ Op 4-4-2011 11:51, Tim Deegan schreef:> At 07:47 +0100 on 04 Apr (1301903251), Jeroen Groenewegen van der Weyden wrote: >> Hello, >> >> I have A question about nested virtualisation, the xen user list >> suggested to post the question here. >> I need nested virtualisation to run an application on out en >> environment, which uses kvm it-self. >> >> I suppose I have to apply these patches: >> - >> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results >> - or http://xenbits.xen.org/ext/cegger/nestedhvm/ >> >> Is it correct these patches are now in the xen-unstable branch? > Not just yet. There were two problems outstanding with patch #12 of the > latest version (a TLB flush race and a subsequent deadlock in the fix), > which are both fixed. As soon as Christoph posts his fixes, the patch > series will be applied. > > Cheers, > > Tim. > >> Anything else to get me on my way. >> >> >> My goal is >> >> DOM0 >> openSUSE 11.4 with xen 4.X >> >> DOMU1 >> SLES 11 SP1 >> kvm guest HVM domu1a (propetary OS) >> kvm guest HVM domu1b (sles 10 sp3) >> kvm guest HVM domu1c (sles 10 sp3) >> kvm guest HVM domu1b (sles 10 sp3) >> >> Op 3-4-2011 23:29, Todd Deshane schreef: >>> On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden >>> <groen692@grosc.com> wrote: >>>> I now am in the position to start testing your suggestion. >>>> >>>> An additional question, is it correct the patches you mention here are part >>>> of the xen4.2-unstable branch? >>>> are these patches for vt-x or for svm only? the developer seems to be from >>>> amd. My machines are running on intel. >>>> >>> The latest patch series is here: >>> http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview-td3415672.html >>> >>> Not sure if it has been accepted yet or not. You can email the >>> developer and/or devel list for confirmation. >>> >>> Thanks, >>> Todd >>> >>>> mfg, >>>> Jeroen >>>> >>>> Op 15-2-2011 16:21, Todd Deshane schreef: >>>>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden >>>>> <groen692@grosc.com> wrote: >>>>>> The why is a good question, >>>>>> We use xen as out main virtualisation platform. But now there are >>>>>> applications that use virtualisation it self. In my case I have an >>>>>> application that uses kvm. So to put it on our xen server it needs to be >>>>>> nested. I guess in the future more of these situation will appear. >>>>>> >>>>>> I read somethings about nested virtualisation and I wonder If somebody >>>>>> has >>>>>> any experience with it. >>>>>> >>>>> To run KVM inside of a guest you need virtualization support within >>>>> the guest (running nested xen doesn''t require the double nested >>>>> (unless you want to do HVM within the nested of course)) >>>>> >>>>> Patches can be found here: >>>>> >>>>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid:z33lihxnbdjghua5+state:results >>>>> >>>>> I believe KVM should also have support for nested vmx and nested svm. >>>>> >>>>> The use cases are fairly slim, so finding users/testers is often >>>>> difficult. >>>>> >>>>> Please report any bugs you find to the appropriate devel lists and cc >>>>> the appropriate nested virtualization maintainers . >>>>> >>>>> Hope that helps. >>>>> >>>>> Thanks, >>>>> Todd >>>>> >>>>> _______________________________________________ >>>>> Xen-users mailing list >>>>> Xen-users@lists.xensource.com >>>>> http://lists.xensource.com/xen-users >>>>> >>> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2011-Apr-07 07:09 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
On 07/04/2011 06:36, "Jeroen Groenewegen van der Weyden" <groen692@grosc.com> wrote:> I couldn''t help noticing some patches were added in the > /staging/xen-unstable branch. So I gave it a go. > > I know it is not called the unstable branch for nothing but, but I want > to let you know that it the entire system crashes when any domain is > created: > xl create someDomain -> results in immediate system crashProbably the same as this, happening often in the automated tests: Apr 7 03:09:18.695893 (XEN) hvm_vcpu_initialise: nestedhvm_vcpu_initialise returned -95 Apr 7 03:09:18.703835 (XEN) Assertion ''d->arch.paging.hap.total_pages == 0'' failed at hap.c:699 [...] Apr 7 03:09:18.915908 (XEN) Xen call trace: Apr 7 03:09:18.915950 (XEN) [<ffff82c48020429f>] hap_teardown+0x1a8/0x229 Apr 7 03:09:18.923861 (XEN) [<ffff82c4801ccb7c>] paging_teardown+0x25/0x135 Apr 7 03:09:18.923922 (XEN) [<ffff82c4801584d1>] domain_relinquish_resources+0x6b/0x23c Apr 7 03:09:18.935903 (XEN) [<ffff82c480105287>] domain_kill+0x95/0xe9 Apr 7 03:09:18.943841 (XEN) [<ffff82c48010324c>] do_domctl+0xd8e/0x1a02 Apr 7 03:09:18.943900 (XEN) [<ffff82c480212f28>] syscall_enter+0xc8/0x122 Apr 7 03:09:18.955837 (XEN) Apr 7 03:09:18.955875 (XEN) Apr 7 03:09:18.955909 (XEN) **************************************** Apr 7 03:09:18.955961 (XEN) Panic on CPU 7: Apr 7 03:09:18.956003 (XEN) Assertion ''d->arch.paging.hap.total_pages == 0'' failed at hap.c:699 Apr 7 03:09:18.963863 (XEN) ****************************************> System: > - domO system openSUSE 11.4 x86_64. kernel 2.6.37.1-1.2-xen > - xen build from staging/xen-unstable (c/s 23172) > - proc intel q9450 > > > +++++++++++++console output > hydra:/etc/xen/vm # xl create groscWindowsXpWorkstation > Parsing config file groscWindowsXpWorkstation > xc: info: VIRTUAL MEMORY ARRANGEMENT: > Loader: 0000000000100000->00000000001795b0 > TOTAL: 0000000000000000->0000000017800000 > ENTRY ADDRESS: 0000000000100000 > xc: info: PHYSICAL MEMORY ALLOCATION: > 4KB PAGES: 0x0000000000000200 > 2MB PAGES: 0x00000000000000bb > 1GB PAGES: 0x0000000000000000 > libxl: error: libxl_device.c:475:libxl__wait_for_device_model Device > Model not ready > xl: fatal error: libxl_create.c:532, rc=-1: > libxl__confirm_device_model_startup > > +++++++++++++ > > Op 4-4-2011 11:51, Tim Deegan schreef: >> At 07:47 +0100 on 04 Apr (1301903251), Jeroen Groenewegen van der Weyden >> wrote: >>> Hello, >>> >>> I have A question about nested virtualisation, the xen user list >>> suggested to post the question here. >>> I need nested virtualisation to run an application on out en >>> environment, which uses kvm it-self. >>> >>> I suppose I have to apply these patches: >>> - >>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mid >>> :z33lihxnbdjghua5+state:results >>> - or http://xenbits.xen.org/ext/cegger/nestedhvm/ >>> >>> Is it correct these patches are now in the xen-unstable branch? >> Not just yet. There were two problems outstanding with patch #12 of the >> latest version (a TLB flush race and a subsequent deadlock in the fix), >> which are both fixed. As soon as Christoph posts his fixes, the patch >> series will be applied. >> >> Cheers, >> >> Tim. >> >>> Anything else to get me on my way. >>> >>> >>> My goal is >>> >>> DOM0 >>> openSUSE 11.4 with xen 4.X >>> >>> DOMU1 >>> SLES 11 SP1 >>> kvm guest HVM domu1a (propetary OS) >>> kvm guest HVM domu1b (sles 10 sp3) >>> kvm guest HVM domu1c (sles 10 sp3) >>> kvm guest HVM domu1b (sles 10 sp3) >>> >>> Op 3-4-2011 23:29, Todd Deshane schreef: >>>> On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden >>>> <groen692@grosc.com> wrote: >>>>> I now am in the position to start testing your suggestion. >>>>> >>>>> An additional question, is it correct the patches you mention here are >>>>> part >>>>> of the xen4.2-unstable branch? >>>>> are these patches for vt-x or for svm only? the developer seems to be from >>>>> amd. My machines are running on intel. >>>>> >>>> The latest patch series is here: >>>> http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overview >>>> -td3415672.html >>>> >>>> Not sure if it has been accepted yet or not. You can email the >>>> developer and/or devel list for confirmation. >>>> >>>> Thanks, >>>> Todd >>>> >>>>> mfg, >>>>> Jeroen >>>>> >>>>> Op 15-2-2011 16:21, Todd Deshane schreef: >>>>>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden >>>>>> <groen692@grosc.com> wrote: >>>>>>> The why is a good question, >>>>>>> We use xen as out main virtualisation platform. But now there are >>>>>>> applications that use virtualisation it self. In my case I have an >>>>>>> application that uses kvm. So to put it on our xen server it needs to be >>>>>>> nested. I guess in the future more of these situation will appear. >>>>>>> >>>>>>> I read somethings about nested virtualisation and I wonder If somebody >>>>>>> has >>>>>>> any experience with it. >>>>>>> >>>>>> To run KVM inside of a guest you need virtualization support within >>>>>> the guest (running nested xen doesn''t require the double nested >>>>>> (unless you want to do HVM within the nested of course)) >>>>>> >>>>>> Patches can be found here: >>>>>> >>>>>> http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+ >>>>>> mid:z33lihxnbdjghua5+state:results >>>>>> >>>>>> I believe KVM should also have support for nested vmx and nested svm. >>>>>> >>>>>> The use cases are fairly slim, so finding users/testers is often >>>>>> difficult. >>>>>> >>>>>> Please report any bugs you find to the appropriate devel lists and cc >>>>>> the appropriate nested virtualization maintainers . >>>>>> >>>>>> Hope that helps. >>>>>> >>>>>> Thanks, >>>>>> Todd >>>>>> >>>>>> _______________________________________________ >>>>>> Xen-users mailing list >>>>>> Xen-users@lists.xensource.com >>>>>> http://lists.xensource.com/xen-users >>>>>> >>>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2011-Apr-07 07:14 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
On 07/04/2011 08:09, "Keir Fraser" <keir.xen@gmail.com> wrote:> On 07/04/2011 06:36, "Jeroen Groenewegen van der Weyden" > <groen692@grosc.com> wrote: > >> I couldn''t help noticing some patches were added in the >> /staging/xen-unstable branch. So I gave it a go. >> >> I know it is not called the unstable branch for nothing but, but I want >> to let you know that it the entire system crashes when any domain is >> created: >> xl create someDomain -> results in immediate system crash > > Probably the same as this, happening often in the automated tests:Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in hvm_vcpu_initialise() even though nestedhvm has not been configured for this domain. This then fails because there is no nestedhvm handler for Intel yet. And beyond that it''s most likely an issue of a broken failure path, or a problem in the HAP code with cleaning up a half-created guest. So it''s probably two bugs. :-) -- Keir> Apr 7 03:09:18.695893 (XEN) hvm_vcpu_initialise: nestedhvm_vcpu_initialise > returned -95 > Apr 7 03:09:18.703835 (XEN) Assertion ''d->arch.paging.hap.total_pages == 0'' > failed at hap.c:699 > [...] > Apr 7 03:09:18.915908 (XEN) Xen call trace: > Apr 7 03:09:18.915950 (XEN) [<ffff82c48020429f>] > hap_teardown+0x1a8/0x229 > Apr 7 03:09:18.923861 (XEN) [<ffff82c4801ccb7c>] > paging_teardown+0x25/0x135 > Apr 7 03:09:18.923922 (XEN) [<ffff82c4801584d1>] > domain_relinquish_resources+0x6b/0x23c > Apr 7 03:09:18.935903 (XEN) [<ffff82c480105287>] domain_kill+0x95/0xe9 > Apr 7 03:09:18.943841 (XEN) [<ffff82c48010324c>] do_domctl+0xd8e/0x1a02 > Apr 7 03:09:18.943900 (XEN) [<ffff82c480212f28>] > syscall_enter+0xc8/0x122 > Apr 7 03:09:18.955837 (XEN) > Apr 7 03:09:18.955875 (XEN) > Apr 7 03:09:18.955909 (XEN) **************************************** > Apr 7 03:09:18.955961 (XEN) Panic on CPU 7: > Apr 7 03:09:18.956003 (XEN) Assertion ''d->arch.paging.hap.total_pages == 0'' > failed at hap.c:699 > Apr 7 03:09:18.963863 (XEN) **************************************** > > >> System: >> - domO system openSUSE 11.4 x86_64. kernel 2.6.37.1-1.2-xen >> - xen build from staging/xen-unstable (c/s 23172) >> - proc intel q9450 >> >> >> +++++++++++++console output >> hydra:/etc/xen/vm # xl create groscWindowsXpWorkstation >> Parsing config file groscWindowsXpWorkstation >> xc: info: VIRTUAL MEMORY ARRANGEMENT: >> Loader: 0000000000100000->00000000001795b0 >> TOTAL: 0000000000000000->0000000017800000 >> ENTRY ADDRESS: 0000000000100000 >> xc: info: PHYSICAL MEMORY ALLOCATION: >> 4KB PAGES: 0x0000000000000200 >> 2MB PAGES: 0x00000000000000bb >> 1GB PAGES: 0x0000000000000000 >> libxl: error: libxl_device.c:475:libxl__wait_for_device_model Device >> Model not ready >> xl: fatal error: libxl_create.c:532, rc=-1: >> libxl__confirm_device_model_startup >> >> +++++++++++++ >> >> Op 4-4-2011 11:51, Tim Deegan schreef: >>> At 07:47 +0100 on 04 Apr (1301903251), Jeroen Groenewegen van der Weyden >>> wrote: >>>> Hello, >>>> >>>> I have A question about nested virtualisation, the xen user list >>>> suggested to post the question here. >>>> I need nested virtualisation to run an application on out en >>>> environment, which uses kvm it-self. >>>> >>>> I suppose I have to apply these patches: >>>> - >>>>http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1+mi>>>> d>>>> :z33lihxnbdjghua5+state:results >>>> - or http://xenbits.xen.org/ext/cegger/nestedhvm/ >>>> >>>> Is it correct these patches are now in the xen-unstable branch? >>> Not just yet. There were two problems outstanding with patch #12 of the >>> latest version (a TLB flush race and a subsequent deadlock in the fix), >>> which are both fixed. As soon as Christoph posts his fixes, the patch >>> series will be applied. >>> >>> Cheers, >>> >>> Tim. >>> >>>> Anything else to get me on my way. >>>> >>>> >>>> My goal is >>>> >>>> DOM0 >>>> openSUSE 11.4 with xen 4.X >>>> >>>> DOMU1 >>>> SLES 11 SP1 >>>> kvm guest HVM domu1a (propetary OS) >>>> kvm guest HVM domu1b (sles 10 sp3) >>>> kvm guest HVM domu1c (sles 10 sp3) >>>> kvm guest HVM domu1b (sles 10 sp3) >>>> >>>> Op 3-4-2011 23:29, Todd Deshane schreef: >>>>> On Sun, Apr 3, 2011 at 8:32 PM, Jeroen Groenewegen van der Weyden >>>>> <groen692@grosc.com> wrote: >>>>>> I now am in the position to start testing your suggestion. >>>>>> >>>>>> An additional question, is it correct the patches you mention here are >>>>>> part >>>>>> of the xen4.2-unstable branch? >>>>>> are these patches for vt-x or for svm only? the developer seems to be >>>>>> from >>>>>> amd. My machines are running on intel. >>>>>> >>>>> The latest patch series is here: >>>>>http://xen.1045712.n5.nabble.com/PATCH-00-12-Nested-Virtualization-Overvie>>>>> w>>>>> -td3415672.html >>>>> >>>>> Not sure if it has been accepted yet or not. You can email the >>>>> developer and/or devel list for confirmation. >>>>> >>>>> Thanks, >>>>> Todd >>>>> >>>>>> mfg, >>>>>> Jeroen >>>>>> >>>>>> Op 15-2-2011 16:21, Todd Deshane schreef: >>>>>>> On Tue, Feb 15, 2011 at 6:47 AM, Jeroen Groenewegen van der Weyden >>>>>>> <groen692@grosc.com> wrote: >>>>>>>> The why is a good question, >>>>>>>> We use xen as out main virtualisation platform. But now there are >>>>>>>> applications that use virtualisation it self. In my case I have an >>>>>>>> application that uses kvm. So to put it on our xen server it needs to >>>>>>>> be >>>>>>>> nested. I guess in the future more of these situation will appear. >>>>>>>> >>>>>>>> I read somethings about nested virtualisation and I wonder If somebody >>>>>>>> has >>>>>>>> any experience with it. >>>>>>>> >>>>>>> To run KVM inside of a guest you need virtualization support within >>>>>>> the guest (running nested xen doesn''t require the double nested >>>>>>> (unless you want to do HVM within the nested of course)) >>>>>>> >>>>>>> Patches can be found here: >>>>>>> >>>>>>>http://xen.markmail.org/search/?q=nested%20vmx#query:nested%20vmx+page:1>>>>>>> +>>>>>>> mid:z33lihxnbdjghua5+state:results >>>>>>> >>>>>>> I believe KVM should also have support for nested vmx and nested svm. >>>>>>> >>>>>>> The use cases are fairly slim, so finding users/testers is often >>>>>>> difficult. >>>>>>> >>>>>>> Please report any bugs you find to the appropriate devel lists and cc >>>>>>> the appropriate nested virtualization maintainers . >>>>>>> >>>>>>> Hope that helps. >>>>>>> >>>>>>> Thanks, >>>>>>> Todd >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Xen-users mailing list >>>>>>> Xen-users@lists.xensource.com >>>>>>> http://lists.xensource.com/xen-users >>>>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xensource.com >>>> http://lists.xensource.com/xen-devel >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christoph Egger
2011-Apr-07 09:04 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
On 04/07/11 07:36, Jeroen Groenewegen van der Weyden wrote:> I couldn''t help noticing some patches were added in the > /staging/xen-unstable branch. So I gave it a go. > > I know it is not called the unstable branch for nothing but, but I want > to let you know that it the entire system crashes when any domain is > created: > xl create someDomain -> results in immediate system crashCan you give the backtrace output, please ? Christoph> System: > - domO system openSUSE 11.4 x86_64. kernel 2.6.37.1-1.2-xen > - xen build from staging/xen-unstable (c/s 23172) > - proc intel q9450 > > > +++++++++++++console output > hydra:/etc/xen/vm # xl create groscWindowsXpWorkstation > Parsing config file groscWindowsXpWorkstation > xc: info: VIRTUAL MEMORY ARRANGEMENT: > Loader: 0000000000100000->00000000001795b0 > TOTAL: 0000000000000000->0000000017800000 > ENTRY ADDRESS: 0000000000100000 > xc: info: PHYSICAL MEMORY ALLOCATION: > 4KB PAGES: 0x0000000000000200 > 2MB PAGES: 0x00000000000000bb > 1GB PAGES: 0x0000000000000000 > libxl: error: libxl_device.c:475:libxl__wait_for_device_model Device > Model not ready > xl: fatal error: libxl_create.c:532, rc=-1: > libxl__confirm_device_model_startup > > +++++++++++++ >-- ---to satisfy European Law for business letters: Advanced Micro Devices GmbH Einsteinring 24, 85689 Dornach b. Muenchen Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Christoph Egger
2011-Apr-07 09:09 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
On 04/07/11 09:09, Keir Fraser wrote:> On 07/04/2011 06:36, "Jeroen Groenewegen van der Weyden" > <groen692@grosc.com> wrote: > >> I couldn''t help noticing some patches were added in the >> /staging/xen-unstable branch. So I gave it a go. >> >> I know it is not called the unstable branch for nothing but, but I want >> to let you know that it the entire system crashes when any domain is >> created: >> xl create someDomain -> results in immediate system crash > > Probably the same as this, happening often in the automated tests: > > Apr 7 03:09:18.695893 (XEN) hvm_vcpu_initialise: nestedhvm_vcpu_initialise > returned -95 > Apr 7 03:09:18.703835 (XEN) Assertion ''d->arch.paging.hap.total_pages == 0'' > failed at hap.c:699 > [...] > Apr 7 03:09:18.915908 (XEN) Xen call trace: > Apr 7 03:09:18.915950 (XEN) [<ffff82c48020429f>] > hap_teardown+0x1a8/0x229 > Apr 7 03:09:18.923861 (XEN) [<ffff82c4801ccb7c>] > paging_teardown+0x25/0x135 > Apr 7 03:09:18.923922 (XEN) [<ffff82c4801584d1>] > domain_relinquish_resources+0x6b/0x23c > Apr 7 03:09:18.935903 (XEN) [<ffff82c480105287>] domain_kill+0x95/0xe9 > Apr 7 03:09:18.943841 (XEN) [<ffff82c48010324c>] do_domctl+0xd8e/0x1a02 > Apr 7 03:09:18.943900 (XEN) [<ffff82c480212f28>] > syscall_enter+0xc8/0x122 > Apr 7 03:09:18.955837 (XEN) > Apr 7 03:09:18.955875 (XEN) > Apr 7 03:09:18.955909 (XEN) **************************************** > Apr 7 03:09:18.955961 (XEN) Panic on CPU 7: > Apr 7 03:09:18.956003 (XEN) Assertion ''d->arch.paging.hap.total_pages == 0'' > failed at hap.c:699 > Apr 7 03:09:18.963863 (XEN) **************************************** >This seems to happen on guest destroy. Which value has d->arch.paging.hap.total_pages ? Christoph -- ---to satisfy European Law for business letters: Advanced Micro Devices GmbH Einsteinring 24, 85689 Dornach b. Muenchen Geschaeftsfuehrer: Alberto Bozzo, Andrew Bowd Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2011-Apr-07 09:30 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
On 07/04/2011 10:09, "Christoph Egger" <Christoph.Egger@amd.com> wrote:> This seems to happen on guest destroy.The original cause is nestedhvm_vcpu_initialise() being unconditionally called on the create path, and always failing on Intel. The eventual crash is a secondary, latent bug I suspect. -- Keir> Which value has d->arch.paging.hap.total_pages ?_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeroen Groenewegen van der Weyden
2011-Apr-07 10:15 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
I''ll be glad to do that, Only I''m really new to this, please give me information on how to gather this information. mfg, Jeroen Op 7-4-2011 11:04, Christoph Egger schreef:> On 04/07/11 07:36, Jeroen Groenewegen van der Weyden wrote: >> I couldn''t help noticing some patches were added in the >> /staging/xen-unstable branch. So I gave it a go. >> >> I know it is not called the unstable branch for nothing but, but I want >> to let you know that it the entire system crashes when any domain is >> created: >> xl create someDomain -> results in immediate system crash > > Can you give the backtrace output, please ? > > Christoph > > >> System: >> - domO system openSUSE 11.4 x86_64. kernel 2.6.37.1-1.2-xen >> - xen build from staging/xen-unstable (c/s 23172) >> - proc intel q9450 >> >> >> +++++++++++++console output >> hydra:/etc/xen/vm # xl create groscWindowsXpWorkstation >> Parsing config file groscWindowsXpWorkstation >> xc: info: VIRTUAL MEMORY ARRANGEMENT: >> Loader: 0000000000100000->00000000001795b0 >> TOTAL: 0000000000000000->0000000017800000 >> ENTRY ADDRESS: 0000000000100000 >> xc: info: PHYSICAL MEMORY ALLOCATION: >> 4KB PAGES: 0x0000000000000200 >> 2MB PAGES: 0x00000000000000bb >> 1GB PAGES: 0x0000000000000000 >> libxl: error: libxl_device.c:475:libxl__wait_for_device_model Device >> Model not ready >> xl: fatal error: libxl_create.c:532, rc=-1: >> libxl__confirm_device_model_startup >> >> +++++++++++++ >> > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tim Deegan
2011-Apr-07 10:29 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote:> On 07/04/2011 08:09, "Keir Fraser" <keir.xen@gmail.com> wrote: > > Probably the same as this, happening often in the automated tests: > > Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in > hvm_vcpu_initialise() even though nestedhvm has not been configured for this > domain. This then fails because there is no nestedhvm handler for Intel yet.This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging tree near you at any moment.> And beyond that it''s most likely an issue of a broken failure path, or a > problem in the HAP code with cleaning up a half-created guest.This one I''ll look into now. Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, Xen Platform Team Citrix Systems UK Ltd. (Company #02937203, SL9 0BG) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeroen Groenewegen van der Weyden
2011-Apr-09 05:07 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
The result with c/s23197
Regular domain now start, en stop normaly, (xl seems to have a problem
though, after domain closes a domain null remains in de xl list/ xmdoe
not have this.)
for nestedVM I added the "nestedhvm=1" in the config file of the
domain.
but the domain does not start because of "invalid argument"
+++++++++ beging console
hydra:/etc/xen/vm # xm create groscNestedVM
Using config file "./groscNestedVM".
Error: (22, ''Invalid argument'')
+++++++++ end console
+++++++++ begin xend.log
hydra:/etc/xen/vm # tail /var/log/xen/xend.log
dominfo = XendDomainInfo.create(config)
File
"/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
line 106, in create
vm.start()
File
"/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
line 473, in start
XendTask.log_progress(0, 30, self._constructDomain)
File "/usr/lib64/python2.7/site-packages/xen/xend/XendTask.py",
line
209, in log_progress
retval = func(*args, **kwds)
File
"/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py",
line 2598, in _constructDomain
xc.hvm_set_param(self.domid, HVM_PARAM_NESTEDHVM, long(nestedhvm))
Error: (22, ''Invalid argument'')
+++++++++++ end xend.log
Op 7-4-2011 12:29, Tim Deegan schreef:> At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote:
>> On 07/04/2011 08:09, "Keir Fraser"<keir.xen@gmail.com>
wrote:
>>> Probably the same as this, happening often in the automated tests:
>> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally
in
>> hvm_vcpu_initialise() even though nestedhvm has not been configured for
this
>> domain. This then fails because there is no nestedhvm handler for Intel
yet.
> This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging
> tree near you at any moment.
>
>> And beyond that it''s most likely an issue of a broken failure
path, or a
>> problem in the HAP code with cleaning up a half-created guest.
> This one I''ll look into now.
>
> Tim.
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2011-Apr-09 06:22 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
On 09/04/2011 06:07, "Jeroen Groenewegen van der Weyden" <groen692@grosc.com> wrote:> The result with c/s23197 > > Regular domain now start, en stop normaly, (xl seems to have a problem > though, after domain closes a domain null remains in de xl list/ xmdoe > not have this.) > for nestedVM I added the "nestedhvm=1" in the config file of the domain. > but the domain does not start because of "invalid argument"NestedHVM currently requires you to be using an AMD processor with hardware nested paging (known as HAP by Xen). Most likely you need to add hap=1 to your config file (and hope your processor supports it). -- Keir> +++++++++ beging console > hydra:/etc/xen/vm # xm create groscNestedVM > Using config file "./groscNestedVM". > Error: (22, ''Invalid argument'') > +++++++++ end console > > +++++++++ begin xend.log > hydra:/etc/xen/vm # tail /var/log/xen/xend.log > dominfo = XendDomainInfo.create(config) > File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", > line 106, in create > vm.start() > File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", > line 473, in start > XendTask.log_progress(0, 30, self._constructDomain) > File "/usr/lib64/python2.7/site-packages/xen/xend/XendTask.py", line > 209, in log_progress > retval = func(*args, **kwds) > File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", > line 2598, in _constructDomain > xc.hvm_set_param(self.domid, HVM_PARAM_NESTEDHVM, long(nestedhvm)) > Error: (22, ''Invalid argument'') > +++++++++++ end xend.log > > > Op 7-4-2011 12:29, Tim Deegan schreef: >> At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote: >>> On 07/04/2011 08:09, "Keir Fraser"<keir.xen@gmail.com> wrote: >>>> Probably the same as this, happening often in the automated tests: >>> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in >>> hvm_vcpu_initialise() even though nestedhvm has not been configured for this >>> domain. This then fails because there is no nestedhvm handler for Intel yet. >> This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging >> tree near you at any moment. >> >>> And beyond that it''s most likely an issue of a broken failure path, or a >>> problem in the HAP code with cleaning up a half-created guest. >> This one I''ll look into now. >> >> Tim. >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeroen Groenewegen van der Weyden
2011-Apr-09 10:18 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
The HAP is default set to one, I check this also with a dry run. Please confirm all this support just amd, because I only have Intel overhere. mfg, Jeroen Op 9-4-2011 8:22, Keir Fraser schreef:> On 09/04/2011 06:07, "Jeroen Groenewegen van der Weyden" > <groen692@grosc.com> wrote: > >> The result with c/s23197 >> >> Regular domain now start, en stop normaly, (xl seems to have a problem >> though, after domain closes a domain null remains in de xl list/ xmdoe >> not have this.) >> for nestedVM I added the "nestedhvm=1" in the config file of the domain. >> but the domain does not start because of "invalid argument" > NestedHVM currently requires you to be using an AMD processor with hardware > nested paging (known as HAP by Xen). Most likely you need to add hap=1 to > your config file (and hope your processor supports it). > > -- Keir > >> +++++++++ beging console >> hydra:/etc/xen/vm # xm create groscNestedVM >> Using config file "./groscNestedVM". >> Error: (22, ''Invalid argument'') >> +++++++++ end console >> >> +++++++++ begin xend.log >> hydra:/etc/xen/vm # tail /var/log/xen/xend.log >> dominfo = XendDomainInfo.create(config) >> File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", >> line 106, in create >> vm.start() >> File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", >> line 473, in start >> XendTask.log_progress(0, 30, self._constructDomain) >> File "/usr/lib64/python2.7/site-packages/xen/xend/XendTask.py", line >> 209, in log_progress >> retval = func(*args, **kwds) >> File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", >> line 2598, in _constructDomain >> xc.hvm_set_param(self.domid, HVM_PARAM_NESTEDHVM, long(nestedhvm)) >> Error: (22, ''Invalid argument'') >> +++++++++++ end xend.log >> >> >> Op 7-4-2011 12:29, Tim Deegan schreef: >>> At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote: >>>> On 07/04/2011 08:09, "Keir Fraser"<keir.xen@gmail.com> wrote: >>>>> Probably the same as this, happening often in the automated tests: >>>> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in >>>> hvm_vcpu_initialise() even though nestedhvm has not been configured for this >>>> domain. This then fails because there is no nestedhvm handler for Intel yet. >>> This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging >>> tree near you at any moment. >>> >>>> And beyond that it''s most likely an issue of a broken failure path, or a >>>> problem in the HAP code with cleaning up a half-created guest. >>> This one I''ll look into now. >>> >>> Tim. >>> >>> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2011-Apr-09 11:15 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
On 09/04/2011 11:18, "Jeroen Groenewegen van der Weyden" <groen692@grosc.com> wrote:> The HAP is default set to one, I check this also with a dry run. > Please confirm all this support just amd, because I only have Intel > overhere.Well, quite apart from being AMD only right now (Intel patches have been posted in the past, but they''re not in tree yet, and I very much doubt old patches would directly apply to our current development tree), I''m also pretty sure that your machine doesn''t support Intel EPT -- if the processor can''t support HAP then we silently discard that configuration request when creating a domain. That''s why you get error 22 (EINVAL). But even if you did have Intel EPT, because Intel''s nested virt patches aren''t in tree yet you''d just get error 95 (EOPNOTSUPP) instead! By the way you can tell if your system supports EPT by grepping in ''xm dmesg'' output (i.e., Xen''s boot log) for a line stating "Extended Page Tables (EPT)". -- Keir> mfg, > Jeroen > > Op 9-4-2011 8:22, Keir Fraser schreef: >> On 09/04/2011 06:07, "Jeroen Groenewegen van der Weyden" >> <groen692@grosc.com> wrote: >> >>> The result with c/s23197 >>> >>> Regular domain now start, en stop normaly, (xl seems to have a problem >>> though, after domain closes a domain null remains in de xl list/ xmdoe >>> not have this.) >>> for nestedVM I added the "nestedhvm=1" in the config file of the domain. >>> but the domain does not start because of "invalid argument" >> NestedHVM currently requires you to be using an AMD processor with hardware >> nested paging (known as HAP by Xen). Most likely you need to add hap=1 to >> your config file (and hope your processor supports it). >> >> -- Keir >> >>> +++++++++ beging console >>> hydra:/etc/xen/vm # xm create groscNestedVM >>> Using config file "./groscNestedVM". >>> Error: (22, ''Invalid argument'') >>> +++++++++ end console >>> >>> +++++++++ begin xend.log >>> hydra:/etc/xen/vm # tail /var/log/xen/xend.log >>> dominfo = XendDomainInfo.create(config) >>> File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", >>> line 106, in create >>> vm.start() >>> File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", >>> line 473, in start >>> XendTask.log_progress(0, 30, self._constructDomain) >>> File "/usr/lib64/python2.7/site-packages/xen/xend/XendTask.py", line >>> 209, in log_progress >>> retval = func(*args, **kwds) >>> File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", >>> line 2598, in _constructDomain >>> xc.hvm_set_param(self.domid, HVM_PARAM_NESTEDHVM, long(nestedhvm)) >>> Error: (22, ''Invalid argument'') >>> +++++++++++ end xend.log >>> >>> >>> Op 7-4-2011 12:29, Tim Deegan schreef: >>>> At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote: >>>>> On 07/04/2011 08:09, "Keir Fraser"<keir.xen@gmail.com> wrote: >>>>>> Probably the same as this, happening often in the automated tests: >>>>> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in >>>>> hvm_vcpu_initialise() even though nestedhvm has not been configured for >>>>> this >>>>> domain. This then fails because there is no nestedhvm handler for Intel >>>>> yet. >>>> This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging >>>> tree near you at any moment. >>>> >>>>> And beyond that it''s most likely an issue of a broken failure path, or a >>>>> problem in the HAP code with cleaning up a half-created guest. >>>> This one I''ll look into now. >>>> >>>> Tim. >>>> >>>> >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeroen Groenewegen van der Weyden
2011-Apr-09 13:32 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
I''m now testing on a intel q9450, which does not seem to have this. I will move my testing to a intel e5645 which seems to have this capability. Question then is, when will the intel support be implemented? is there commitment from some developer? is there a roadmap planned. in short what is the story here. mfg, Jeroen Op 9-4-2011 13:15, Keir Fraser schreef:> On 09/04/2011 11:18, "Jeroen Groenewegen van der Weyden" > <groen692@grosc.com> wrote: > >> The HAP is default set to one, I check this also with a dry run. >> Please confirm all this support just amd, because I only have Intel >> overhere. > Well, quite apart from being AMD only right now (Intel patches have been > posted in the past, but they''re not in tree yet, and I very much doubt old > patches would directly apply to our current development tree), I''m also > pretty sure that your machine doesn''t support Intel EPT -- if the processor > can''t support HAP then we silently discard that configuration request when > creating a domain. That''s why you get error 22 (EINVAL). But even if you did > have Intel EPT, because Intel''s nested virt patches aren''t in tree yet you''d > just get error 95 (EOPNOTSUPP) instead! > > By the way you can tell if your system supports EPT by grepping in ''xm > dmesg'' output (i.e., Xen''s boot log) for a line stating "Extended Page > Tables (EPT)". > > -- Keir > >> mfg, >> Jeroen >> >> Op 9-4-2011 8:22, Keir Fraser schreef: >>> On 09/04/2011 06:07, "Jeroen Groenewegen van der Weyden" >>> <groen692@grosc.com> wrote: >>> >>>> The result with c/s23197 >>>> >>>> Regular domain now start, en stop normaly, (xl seems to have a problem >>>> though, after domain closes a domain null remains in de xl list/ xmdoe >>>> not have this.) >>>> for nestedVM I added the "nestedhvm=1" in the config file of the domain. >>>> but the domain does not start because of "invalid argument" >>> NestedHVM currently requires you to be using an AMD processor with hardware >>> nested paging (known as HAP by Xen). Most likely you need to add hap=1 to >>> your config file (and hope your processor supports it). >>> >>> -- Keir >>> >>>> +++++++++ beging console >>>> hydra:/etc/xen/vm # xm create groscNestedVM >>>> Using config file "./groscNestedVM". >>>> Error: (22, ''Invalid argument'') >>>> +++++++++ end console >>>> >>>> +++++++++ begin xend.log >>>> hydra:/etc/xen/vm # tail /var/log/xen/xend.log >>>> dominfo = XendDomainInfo.create(config) >>>> File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", >>>> line 106, in create >>>> vm.start() >>>> File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", >>>> line 473, in start >>>> XendTask.log_progress(0, 30, self._constructDomain) >>>> File "/usr/lib64/python2.7/site-packages/xen/xend/XendTask.py", line >>>> 209, in log_progress >>>> retval = func(*args, **kwds) >>>> File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", >>>> line 2598, in _constructDomain >>>> xc.hvm_set_param(self.domid, HVM_PARAM_NESTEDHVM, long(nestedhvm)) >>>> Error: (22, ''Invalid argument'') >>>> +++++++++++ end xend.log >>>> >>>> >>>> Op 7-4-2011 12:29, Tim Deegan schreef: >>>>> At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote: >>>>>> On 07/04/2011 08:09, "Keir Fraser"<keir.xen@gmail.com> wrote: >>>>>>> Probably the same as this, happening often in the automated tests: >>>>>> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally in >>>>>> hvm_vcpu_initialise() even though nestedhvm has not been configured for >>>>>> this >>>>>> domain. This then fails because there is no nestedhvm handler for Intel >>>>>> yet. >>>>> This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging >>>>> tree near you at any moment. >>>>> >>>>>> And beyond that it''s most likely an issue of a broken failure path, or a >>>>>> problem in the HAP code with cleaning up a half-created guest. >>>>> This one I''ll look into now. >>>>> >>>>> Tim. >>>>> >>>>> >>> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2011-Apr-09 14:31 UTC
Re: [Xen-devel] Re: [Xen-users] Nested Virtualisation question
On 09/04/2011 14:32, "Jeroen Groenewegen van der Weyden" <groen692@grosc.com> wrote:> I''m now testing on a intel q9450, which does not seem to have this. I > will move my testing to a intel e5645 which seems to have this capability. > > Question then is, when will the intel support be implemented? is there > commitment from some developer? is there a roadmap planned. in short > what is the story here.It has been implemented already, for example see a previous patchset submission: http://xen.1045712.n5.nabble.com/PATCH-00-16-Nested-virtualization-for-VMX-t d2808075.html It just needs updating to development tip, and in particular sync''ing with the nestedhvm common bits that got applied last week. Someone from Intel would have to speak for how long it will take to schedule and do that work, but I would guess we''re talking weeks, rather than days or months. -- Keir> mfg, > Jeroen > > Op 9-4-2011 13:15, Keir Fraser schreef: >> On 09/04/2011 11:18, "Jeroen Groenewegen van der Weyden" >> <groen692@grosc.com> wrote: >> >>> The HAP is default set to one, I check this also with a dry run. >>> Please confirm all this support just amd, because I only have Intel >>> overhere. >> Well, quite apart from being AMD only right now (Intel patches have been >> posted in the past, but they''re not in tree yet, and I very much doubt old >> patches would directly apply to our current development tree), I''m also >> pretty sure that your machine doesn''t support Intel EPT -- if the processor >> can''t support HAP then we silently discard that configuration request when >> creating a domain. That''s why you get error 22 (EINVAL). But even if you did >> have Intel EPT, because Intel''s nested virt patches aren''t in tree yet you''d >> just get error 95 (EOPNOTSUPP) instead! >> >> By the way you can tell if your system supports EPT by grepping in ''xm >> dmesg'' output (i.e., Xen''s boot log) for a line stating "Extended Page >> Tables (EPT)". >> >> -- Keir >> >>> mfg, >>> Jeroen >>> >>> Op 9-4-2011 8:22, Keir Fraser schreef: >>>> On 09/04/2011 06:07, "Jeroen Groenewegen van der Weyden" >>>> <groen692@grosc.com> wrote: >>>> >>>>> The result with c/s23197 >>>>> >>>>> Regular domain now start, en stop normaly, (xl seems to have a problem >>>>> though, after domain closes a domain null remains in de xl list/ xmdoe >>>>> not have this.) >>>>> for nestedVM I added the "nestedhvm=1" in the config file of the domain. >>>>> but the domain does not start because of "invalid argument" >>>> NestedHVM currently requires you to be using an AMD processor with hardware >>>> nested paging (known as HAP by Xen). Most likely you need to add hap=1 to >>>> your config file (and hope your processor supports it). >>>> >>>> -- Keir >>>> >>>>> +++++++++ beging console >>>>> hydra:/etc/xen/vm # xm create groscNestedVM >>>>> Using config file "./groscNestedVM". >>>>> Error: (22, ''Invalid argument'') >>>>> +++++++++ end console >>>>> >>>>> +++++++++ begin xend.log >>>>> hydra:/etc/xen/vm # tail /var/log/xen/xend.log >>>>> dominfo = XendDomainInfo.create(config) >>>>> File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", >>>>> line 106, in create >>>>> vm.start() >>>>> File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", >>>>> line 473, in start >>>>> XendTask.log_progress(0, 30, self._constructDomain) >>>>> File "/usr/lib64/python2.7/site-packages/xen/xend/XendTask.py", line >>>>> 209, in log_progress >>>>> retval = func(*args, **kwds) >>>>> File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", >>>>> line 2598, in _constructDomain >>>>> xc.hvm_set_param(self.domid, HVM_PARAM_NESTEDHVM, long(nestedhvm)) >>>>> Error: (22, ''Invalid argument'') >>>>> +++++++++++ end xend.log >>>>> >>>>> >>>>> Op 7-4-2011 12:29, Tim Deegan schreef: >>>>>> At 08:14 +0100 on 07 Apr (1302164093), Keir Fraser wrote: >>>>>>> On 07/04/2011 08:09, "Keir Fraser"<keir.xen@gmail.com> wrote: >>>>>>>> Probably the same as this, happening often in the automated tests: >>>>>>> Likely reason: nestedhvm_vcpu_initialise() being called unconditionally >>>>>>> in >>>>>>> hvm_vcpu_initialise() even though nestedhvm has not been configured for >>>>>>> this >>>>>>> domain. This then fails because there is no nestedhvm handler for Intel >>>>>>> yet. >>>>>> This bug should be fixed by 23175:e5a750d1bf9b, arriving at a staging >>>>>> tree near you at any moment. >>>>>> >>>>>>> And beyond that it''s most likely an issue of a broken failure path, or a >>>>>>> problem in the HAP code with cleaning up a half-created guest. >>>>>> This one I''ll look into now. >>>>>> >>>>>> Tim. >>>>>> >>>>>> >>>> >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel