Libguestfs - May 2013 - guestfish runs w/ a nested guest

# Ref: http://libguestfs.org/guestfs-performance.1.html

Run the below command:

    $ time guestfish -a /dev/null run

NOTE: Discard the first few results, to get a hot cache. (Thanks Rich.)

1/ L0. with L1 running.
----------------------------------------------------------------------
    $ for i in {1..10}; do time guestfish -a /dev/null run; done
    real    0m28.277s
    user    0m11.028s
    sys     0m0.706s

    real    0m2.753s
    user    0m0.054s
    sys     0m0.049s

    real    0m2.771s
    user    0m0.063s
    sys     0m0.044s

    real    0m2.774s
    user    0m0.053s
    sys     0m0.044s

    real    0m2.772s
    user    0m0.054s
    sys     0m0.049s

    real    0m2.715s
    user    0m0.051s
    sys     0m0.045s

    real    0m2.753s
    user    0m0.064s
    sys     0m0.037s

    real    0m2.709s
    user    0m0.059s
    sys     0m0.044s

    real    0m2.782s
    user    0m0.055s
    sys     0m0.045s

    real    0m2.775s
    user    0m0.054s
    sys     0m0.049s
----------------------------------------------------------------------

=> Remove the appliance and re-run:
----------------------------------------------------------------------
    $ rm -rf /var/tmp/guestfs-0

    $ for in in {1..2}; do time guestfish -a /dev/null run; done
    real    0m14.623s
    user    0m10.172s
    sys     0m0.520s

    real    0m2.790s
    user    0m0.048s
    sys     0m0.046s
----------------------------------------------------------------------


2. L1 w/ nesting, VMCS shadowing enabled. NOTE: *NO* L2 guests are running
   while this test is being performed.

----------------------------------------------------------------------
    $ for i in {1..10}; do time guestfish -a /dev/null run; done | tee
guestfish-timings-L1.txt
    real    0m55.182s
    user    0m14.747s
    sys     0m8.501s

    real    0m8.754s
    user    0m0.054s
    sys     0m0.494s

    real    0m9.576s
    user    0m0.056s
    sys     0m0.490s

    real    0m8.793s
    user    0m0.056s
    sys     0m0.498s

    real    0m8.767s
    user    0m0.053s
    sys     0m0.492s

    real    0m8.715s
    user    0m0.049s
    sys     0m0.502s

    real    0m8.814s
    user    0m0.051s
    sys     0m0.503s

    real    0m8.765s
    user    0m0.046s
    sys     0m0.516s

    real    0m8.783s
    user    0m0.050s
    sys     0m0.502s

    real    0m8.739s
    user    0m0.054s
    sys     0m0.496s
----------------------------------------------------------------------


=> Next, is to try: L1 w/o nesting, VMCS shadowing disabled.
----------------------------------------------------------------------

     # On L0
     $ vi /etc/modprobe.d/dist.conf
        - s/options kvm-intel nested=y/options kvm-intel nested=n/

     # For the test purpose, remove the XML fragment (which adds VMX
     # extensions) for L1 CPU
     $ virsh edit regular-guest
        - Remove the below fragment, save it:
          <cpu mode='custom' match='exact'>
            <model fallback='allow'>Haswell</model>
            <feature policy='require' name='vmx'/>
          </cpu>

     # Reboot
     $ init 6

     # Check for the nested parameter on L0
     $ cat /sys/module/kvm_intel/parameters/nested
     N

     # Remove any old appliances
     $ rm -rf /var/tmp/.guestfs-0

     # Run:
     $ for i in {1..10}; do time guestfish -a /dev/null run; done | tee \
       guestfish-timings-L1-no-nested.txt
----------------------------------------------------------------------


-- 
/kashyap

> => Next, is to try: L1 w/o nesting, VMCS shadowing disabled.
> ----------------------------------------------------------------------
> 
>      # On L0
>      $ vi /etc/modprobe.d/dist.conf
>         - s/options kvm-intel nested=y/options kvm-intel nested=n/
> 
>      # For the test purpose, remove the XML fragment (which adds VMX
>      # extensions) for L1 CPU
>      $ virsh edit regular-guest
>         - Remove the below fragment, save it:
>           <cpu mode='custom' match='exact'>
>             <model fallback='allow'>Haswell</model>
>             <feature policy='require' name='vmx'/>
>           </cpu>
> 
>      # Reboot
>      $ init 6
> 
>      # Check for the nested parameter on L0
>      $ cat /sys/module/kvm_intel/parameters/nested
>      N
> 
>      # Remove any old appliances
>      $ rm -rf /var/tmp/.guestfs-0
> 
>      # Run:
>      $ for i in {1..10}; do time guestfish -a /dev/null run; done | tee \
>        guestfish-timings-L1-no-nested.txt
> ----------------------------------------------------------------------
> 
Results for the above case -- no nesting, guestfish run in 1st level guest:
----------------------------------------------------------------------
$ for i in {1..10}; do time guestfish -a /dev/null run; done | tee
guestfish-timings-L1-no-nesting.txt


real    1m10.899s
user    0m13.558s
sys     0m8.167s

real    0m29.899s
user    0m0.066s
sys     0m0.522s

real    0m25.310s
user    0m0.056s
sys     0m0.516s

real    0m35.267s
user    0m0.053s
sys     0m0.515s

real    0m24.872s
user    0m0.051s
sys     0m0.545s

real    0m24.827s
user    0m0.044s
sys     0m0.538s

real    0m41.053s
user    0m0.060s
sys     0m0.531s

real    0m29.107s
user    0m0.049s
sys     0m0.534s

real    0m30.196s
user    0m0.051s
sys     0m0.532s

real    0m52.815s
user    0m0.052s
sys     0m0.520s

----------------------------------------------------------------------

To summarize:

L0 (bare-metal):
	- guestfish run: ~ 3 seconds

L1 (guest):
	- w/o nesting, guestfish run: ~ 52 seconds
	- w/ nesting,  guestfish run: ~ 8 seconds  (Almost 6 1/2 times speed
improvement)



For Reference:
--------------
1/ Level-1 guest qemu-kvm command-line, indicating that I *did* remove
<cpu> fragment
gracefully, i.e. no nesting.
==================================
$ ps -ef | grep -i qemu
qemu      2367     1 86 08:29 ?        00:00:03 /usr/bin/qemu-system-x86_64
-machine
accel=kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -m
6144 -smp
4,sockets=4,cores=1,threads=1 -uuid 4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2
-nographic
-no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
-mon
chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
-chardev
pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device
usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
==================================
2/ Level-1 guest qemu-kvm command-line *with* nesting:
==================================$ ps -ef | grep -i qemu
    qemu      4962     1 21 15:41 ?        00:00:41 /usr/bin/qemu-system-x86_64
-machine
accel=kvm -name regular-guest -S -machine pc-i440fx-1.4,accel=kvm,usb=off -cpu
Haswell,+vmx -m 6144 -smp 4,sockets=4,cores=1,threads=1 -uuid
4ed9ac0b-7f72-dfcf-68b3-e6fe2ac588b2 -nographic -no-user-config -nodefaults
-chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/regular-guest.monitor,server,nowait
-mon
chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/home/test/vmimages/regular-guest.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:80:c1:34,bus=pci.0,addr=0x3
-chardev
pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device
usb-tablet,id=input0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
==================================
The setup details are here -
https://raw.github.com/kashyapc/nvmx-haswell/master/SETUP-nVMX.rst

Rich, if you have further thoughts, please let me know.

Thanks.

-- 
/kashyap