CentOS virt - Nov 2015 - Install nested ESXi 6.x host under CentOS 7 kvm host

Hi all,

 I am trying to install nested Esxi 6.x under a Centos7 kvm host to
use it as test lab for new ESXi versions, but I am doing something
wrong because I can't install it.

 I have configured kvm and kvm_intel modules with the following options:

options kvm ignore_msrs=1
options kvm_intel nested=1 ept=1

 and I add the following options in <cpu> xml guest file:

  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Westmere</model>
    <feature policy='require' name='vmx'/>
  </cpu>

qemu-kvm package is installed from Virtualization SIG:

qemu-kvm-common-ev-2.1.2-23.el7_1.8.1.x86_64
qemu-kvm-ev-2.1.2-23.el7_1.8.1.x86_64

 Am I doing something wrong or is it not possible to accomplish this scenario??

Thanks.

On 11/26/15 09:47, C. L. Martinez wrote:
> Hi all,
>
>   I am trying to install nested Esxi 6.x under a Centos7 kvm host to
> use it as test lab for new ESXi versions, but I am doing something
> wrong because I can't install it.
>
>   I have configured kvm and kvm_intel modules with the following options:
>
> options kvm ignore_msrs=1
> options kvm_intel nested=1 ept=1
>
>   and I add the following options in <cpu> xml guest file:
>
>    <cpu mode='custom' match='exact'>
>      <model fallback='allow'>Westmere</model>
>      <feature policy='require' name='vmx'/>
>    </cpu>
>
> qemu-kvm package is installed from Virtualization SIG:
>
> qemu-kvm-common-ev-2.1.2-23.el7_1.8.1.x86_64
> qemu-kvm-ev-2.1.2-23.el7_1.8.1.x86_64
>
>   Am I doing something wrong or is it not possible to accomplish this
scenario??
Here are my old notes how to run ESXi6 on KVM on Fedora 21:

# cat /etc/modprobe.d/nested.conf
options kvm ignore_msrs=1
options kvm-intel nested=y ept=y


Install ESXI6 Hypervisor with the vmxnet3 nic:

# virt-install --connect=qemu:///system -n esxi6 --cpu=host --vcpus=2 
--ram 4096 --os-type=linux 
--cdrom=/var/lib/libvirt/images/VMware-VMvisor-Installer-6.0.0-2494585.x86_64.iso
--disk 
path=/var/lib/libvirt/images/esxi6.qcow2,format=qcow2,bus=ide,size=25,cache=none
--hvm --network=default,model=vmxnet3 --virt-type=kvm --arch=x86_64 
--graphics spice --accelerate --video qxl --channel spicevmc

or with the e1000 nic:

# virt-install --connect=qemu:///system -n esxi6 --cpu=host --vcpus=2 
--ram 4096 --os-type=linux 
--cdrom=/var/lib/libvirt/images/VMware-VMvisor-Installer-6.0.0-2494585.x86_64.iso
--disk 
path=/var/lib/libvirt/images/esxi6.qcow2,format=qcow2,bus=ide,size=25,cache=none
--hvm --network=default,model=e1000 --virt-type=kvm --arch=x86_64 
--graphics spice --accelerate --video qxl --channel spicevmc


Boot the ESXi VM and in the console interface go to Troubleshooting 
Options and enable SSH and Console access.

SSH to the ESXi VM:

$ ssh -o IdentitiesOnly=yes root@<IP of ESXi6 VM>

# cd /etc/vmware
# vi ./config and add the following options:

vmx.allowNested = TRUE
hv.assumeEnabled = TRUE

Add the IP address of the ESXi Hypervisor to /etc/hosts
192.168.122.172 esxi.lab.local

Now reboot the ESXi VM to make those options active.



I was able to install vCenter 6 (VCSA) on ESXi6 with these steps:

Create the template json file which is explained in
http://www.vmware.com/files/pdf/products/vsphere/VMware-vsphere-60-vcenter-server-appliance-cmdline-install.pdf

Some tips:
http://www.vladan.fr/vcenter-appliance-6-0-simplified-scripted-installation/


$ cat /home/patrick/VMware/patrick_embedded.json
{
     "__comments":
     [
         "Test template for VCSA deployment on a nested ESXi Hypervisor 
running on KVM on F21"
     ],

     "deployment":
     {
         "esx.hostname":"esxi.lab.local",
         "esx.datastore":"datastore1",
         "esx.username":"root",
         "esx.password":"password",
         "deployment.option":"tiny",
         "deployment.network":"vm",
         "appliance.name":"vcsa",
         "appliance.thin.disk.mode":true
     },

     "vcsa":
     {
         "system":
         {
             "root.password":"password",
             "ssh.enable":true
         },

         "sso":
         {
             "password":"Password1!",
             "domain-name":"lab.local",
             "site-name":"patrick_lab"
         }

     }
}


Next mount the VMware-VCSA-all-6.0.0-2562643.iso file
# cd /run/media/patrick/VMware\ VCSA/vcsa-cli-installer/lin64

And run the command to deploy the VCSA appliance on top of ESXi:
$ vcsa-deploy -v /home/patrick/VMware/patrick_embedded.json


[patrick at lab lin64]$ ./vcsa-deploy -v 
/home/patrick/VMware/patrick_embedded.json

Start vCSA command line installer to deploy vCSA "vcsa", an embedded
node.

Please see /tmp/vcsa-cli-installer-8159mI.log for logging information.

The SSO password meets the installation requirements.
Opening vCSA image: /run/media/patrick/VMware VCSA/vcsa/vmware-vcsa
SSL fingerprint 
fb:9a:f8:d2:86:14:c0:17:6d:02:f4:9d:da:e8:31:ee:fb:3b:58:d4 is different 
from host fingerprint 
51:A6:F1:28:E7:2D:9B:02:7A:97:0A:1A:87:A3:42:A8:4C:42:24:03 for host 
esxi.lab.local as target type.
Accept update of SSL fingerprint in known host file
Write 'yes' or 'no'
yes
Opening VI target: vi://root at esxi.lab.local:443/
Deploying to VI: vi://root at esxi.lab.local:443/

Progress: 99%
Transfer Completed
Powering on VM: vcsa

Progress: 48%
Power On Completed

Installing services...
Progress: 5%. Setting up storage
Progress: 50%. Installing RPMs
Progress: 55%. Installed VMware-unixODBC-2.3.1.vmw.2-6.0.0.x86_64.rpm
...
Profit!

Or not because the Firefox VCSA plugin did not work on a Linux (Fedora) 
host which you need to actually use vCenter. If you make the VCSA gui 
work with Firefox on Linux, I would appreciate an update.

HTH,
Patrick

On 11/27/2015 01:32 PM, Patrick Laimbock wrote:
> On 11/26/15 09:47, C. L. Martinez wrote:
>> Hi all,
>>
>>   I am trying to install nested Esxi 6.x under a Centos7 kvm host to
>> use it as test lab for new ESXi versions, but I am doing something
>> wrong because I can't install it.
>>
>>   I have configured kvm and kvm_intel modules with the following
options:
>>
>> options kvm ignore_msrs=1
>> options kvm_intel nested=1 ept=1
>>
>>   and I add the following options in <cpu> xml guest file:
>>
>>    <cpu mode='custom' match='exact'>
>>      <model fallback='allow'>Westmere</model>
>>      <feature policy='require' name='vmx'/>
>>    </cpu>
>>
>> qemu-kvm package is installed from Virtualization SIG:
>>
>> qemu-kvm-common-ev-2.1.2-23.el7_1.8.1.x86_64
>> qemu-kvm-ev-2.1.2-23.el7_1.8.1.x86_64
>>
>>   Am I doing something wrong or is it not possible to accomplish this
>> scenario??
>
> Here are my old notes how to run ESXi6 on KVM on Fedora 21:
>
> # cat /etc/modprobe.d/nested.conf
> options kvm ignore_msrs=1
> options kvm-intel nested=y ept=y
>
>
> Install ESXI6 Hypervisor with the vmxnet3 nic:
>
> # virt-install --connect=qemu:///system -n esxi6 --cpu=host --vcpus=2
> --ram 4096 --os-type=linux
>
--cdrom=/var/lib/libvirt/images/VMware-VMvisor-Installer-6.0.0-2494585.x86_64.iso
> --disk
>
path=/var/lib/libvirt/images/esxi6.qcow2,format=qcow2,bus=ide,size=25,cache=none
> --hvm --network=default,model=vmxnet3 --virt-type=kvm --arch=x86_64
> --graphics spice --accelerate --video qxl --channel spicevmc
>
> or with the e1000 nic:
>
> # virt-install --connect=qemu:///system -n esxi6 --cpu=host --vcpus=2
> --ram 4096 --os-type=linux
>
--cdrom=/var/lib/libvirt/images/VMware-VMvisor-Installer-6.0.0-2494585.x86_64.iso
> --disk
>
path=/var/lib/libvirt/images/esxi6.qcow2,format=qcow2,bus=ide,size=25,cache=none
> --hvm --network=default,model=e1000 --virt-type=kvm --arch=x86_64
> --graphics spice --accelerate --video qxl --channel spicevmc
>
>
> Boot the ESXi VM and in the console interface go to Troubleshooting
> Options and enable SSH and Console access.
>
> SSH to the ESXi VM:
>
> $ ssh -o IdentitiesOnly=yes root@<IP of ESXi6 VM>
>
> # cd /etc/vmware
> # vi ./config and add the following options:
>
> vmx.allowNested = TRUE
> hv.assumeEnabled = TRUE
>
> Add the IP address of the ESXi Hypervisor to /etc/hosts
> 192.168.122.172 esxi.lab.local
>
> Now reboot the ESXi VM to make those options active.
>
>
>
> I was able to install vCenter 6 (VCSA) on ESXi6 with these steps:
>
> Create the template json file which is explained in
>
http://www.vmware.com/files/pdf/products/vsphere/VMware-vsphere-60-vcenter-server-appliance-cmdline-install.pdf
>
>
> Some tips:
>
http://www.vladan.fr/vcenter-appliance-6-0-simplified-scripted-installation/
>
>
>
> $ cat /home/patrick/VMware/patrick_embedded.json
> {
>      "__comments":
>      [
>          "Test template for VCSA deployment on a nested ESXi
Hypervisor
> running on KVM on F21"
>      ],
>
>      "deployment":
>      {
>          "esx.hostname":"esxi.lab.local",
>          "esx.datastore":"datastore1",
>          "esx.username":"root",
>          "esx.password":"password",
>          "deployment.option":"tiny",
>          "deployment.network":"vm",
>          "appliance.name":"vcsa",
>          "appliance.thin.disk.mode":true
>      },
>
>      "vcsa":
>      {
>          "system":
>          {
>              "root.password":"password",
>              "ssh.enable":true
>          },
>
>          "sso":
>          {
>              "password":"Password1!",
>              "domain-name":"lab.local",
>              "site-name":"patrick_lab"
>          }
>
>      }
> }
>
>
> Next mount the VMware-VCSA-all-6.0.0-2562643.iso file
> # cd /run/media/patrick/VMware\ VCSA/vcsa-cli-installer/lin64
>
> And run the command to deploy the VCSA appliance on top of ESXi:
> $ vcsa-deploy -v /home/patrick/VMware/patrick_embedded.json
>
>
> [patrick at lab lin64]$ ./vcsa-deploy -v
> /home/patrick/VMware/patrick_embedded.json
>
> Start vCSA command line installer to deploy vCSA "vcsa", an
embedded node.
>
> Please see /tmp/vcsa-cli-installer-8159mI.log for logging information.
>
> The SSO password meets the installation requirements.
> Opening vCSA image: /run/media/patrick/VMware VCSA/vcsa/vmware-vcsa
> SSL fingerprint
> fb:9a:f8:d2:86:14:c0:17:6d:02:f4:9d:da:e8:31:ee:fb:3b:58:d4 is different
> from host fingerprint
> 51:A6:F1:28:E7:2D:9B:02:7A:97:0A:1A:87:A3:42:A8:4C:42:24:03 for host
> esxi.lab.local as target type.
> Accept update of SSL fingerprint in known host file
> Write 'yes' or 'no'
> yes
> Opening VI target: vi://root at esxi.lab.local:443/
> Deploying to VI: vi://root at esxi.lab.local:443/
>
> Progress: 99%
> Transfer Completed
> Powering on VM: vcsa
>
> Progress: 48%
> Power On Completed
>
> Installing services...
> Progress: 5%. Setting up storage
> Progress: 50%. Installing RPMs
> Progress: 55%. Installed VMware-unixODBC-2.3.1.vmw.2-6.0.0.x86_64.rpm
> ...
> Profit!
>
> Or not because the Firefox VCSA plugin did not work on a Linux (Fedora)
> host which you need to actually use vCenter. If you make the VCSA gui
> work with Firefox on Linux, I would appreciate an update.
>
> HTH,
> Patrick
Many thanks Patrick for your valuable help!!. I've found the problem: 
removing my <cpu> entries and adding "<cpu
mode='host-passthrough'>",
works...

But in my case, I can't assign vmxnet3 as a virtual nic for ESXi, it 
seems is not supported at this time ...