Libguestfs - Feb 2014 - Debugging nested KVM guest (L2) booting with libguestfs/gdb

Heya,

With latest Fedora Rawhide Kernel, I see a nested KVM guest hanging at
boot (not unusual). Rich once suggested this[1] to try to attach gdb to
the nested L2 guest to find out where_ it's stuck, tonight I set out to
try it out (with KVM & TCG).

Below is all what I tried.

In guest hypervisor (L1):

    $ git clone git://github.com/libguestfs/libguestfs.git
    $ git log  | head -1
    commit 82a4a8f02c5706979d961ad6c4ac767a37a3a7c9

    # Activate gdb debugging code by turning its
    # conditional compilation directive: s/#if 0/#if 1
    $ vi src/launch.c
    [. . .] # Edit, save it
 
Double ensure the gdb pre-processor directive is turned on:

    $ grep "\-S" src/launch-direct.c -A4 -B2
       */
    #if 1
      ADD_CMDLINE ("-S");
      ADD_CMDLINE ("-s");
      warning (g, "qemu debugging is enabled, connect gdb to tcp::1234 to
begin");
    #endif

Compile:

    $ ./autogen.sh
    $ make -j4
    [. . .] # Compile, address anything that comes up
    $ echo $?
    0

Install Kernel-debug info:

    $ yum install --enablerepo=fedora-debuginfo kernel-debuginfo

Enable the 'direct' backend and invoke the appliance using the `run`
script:

    $ export LIBGUESTFS_BACKEND=direct        
    $ ./run libguestfs-test-tool 

The 'qemu-kvm' command-line just hung at:

    $ ./run libguestfs-test-tool
    [. . .]
        -chardev
socket,path=/home/tuser1/src/libguestfs/tmp/libguestfspCGc1F/guestfsd.sock,id=channel0
\
    -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
    -append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check
lpj=2294686 acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0
guestfs_verbose=1 TERM=screen'


Let's try with TCG
------------------

    $ guestfish get_backend
    direct
    $ export LIBGUESTFS_BACKEND_SETTINGS=force_tcg
    $ guestfish get_backend_settings
    force_tcg

Run the appliance:

    $ ./run libguestfs-test-tool

Again, `qemu-kvm` CLI just hung just like with KVM acceleration case


Try a couple more things
------------------------

- Run `qemu-sanity-check` in L1 -- Of course, this fails too.

- Invoke (from a different shell, as root) QEMU directly with gdb
  debugging options -s -S with KVM on L1:

   $ qemu-system-x86_64 -s -S -nographic -nodefconfig \
     -nodefaults -machine accel=kvm -m 4000 \
     -drive
file=/home/tuser1/vmimages/fedora-20.qcow2,if=ide,format=qcow2,cache=none \
     -serial stdio

  Result: Just hung there.

  Tru with TCG:

   $ qemu-system-x86_64 -s -S -nographic -nodefconfig \
     -nodefaults -machine accel=tcg -m 4000 \
     -drive
file=/home/tuser1/vmimages/fedora-20.qcow2,if=ide,format=qcow2,cache=none \
     -serial stdio

  to no avail: it's just hung there, again.


What else I can try? 

Did I get this right or am I making any elementary mistake here? (I'm yet
to try on a different L1 guest on a different hardware, before that I
just wanted to run this by the list.)


Version info
------------

On L1:

    $ virt-what
    kvm
    
    $ uname -r; rpm -q libvirt-daemon-kvm qemu-system-x86 \
      libguestfs kernel-debuginfo
    3.14.0-0.rc2.git0.1.fc21.x86_64
    libvirt-daemon-kvm-1.2.1-2.fc21.x86_64
    qemu-system-x86-1.7.0-4.fc21.x86_64
    libguestfs-1.25.33-1.fc21.x86_64
    kernel-debuginfo-3.11.10-301.fc20.x86_64


  [1]
https://github.com/libguestfs/libguestfs/blob/master/src/launch-direct.c#L404

-- 
/kashyap

On Thu, Feb 13, 2014 at 12:08:37AM +0530, Kashyap Chamarthy wrote:
 
[. . .]
> 
> Try a couple more things
> ------------------------
> 
> - Run `qemu-sanity-check` in L1 -- Of course, this fails too.
> 
> - Invoke (from a different shell, as root) QEMU directly with gdb
>   debugging options -s -S with KVM on L1:
> 
>    $ qemu-system-x86_64 -s -S -nographic -nodefconfig \
>      -nodefaults -machine accel=kvm -m 4000 \
>      -drive
file=/home/tuser1/vmimages/fedora-20.qcow2,if=ide,format=qcow2,cache=none \
>      -serial stdio
So,  using -serial mon:stdio like below, and using CTL+a to toggles
monitor and stdio (thanks to Amit Shah for the tip):

    $ qemu-system-x86_64 -s -S -nographic -nodefconfig   \
      -nodefaults -enable-kvm -machine accel=kvm -m 4000 \
      -drive file=/home/tuser1/vmimages/fedora-20.qcow2,format=qcow2,cache=none
\
      -device virtio-serial -serial mon:stdio
    
    C-a h    print this help
    C-a x    exit emulator
    C-a s    save disk data back to file (if -snapshot)
    C-a t    toggle console timestamps
    C-a b    send break (magic sysrq)
    C-a c    switch between console and monitor
    C-a C-a  sends C-a
    QEMU 1.7.0 monitor - type 'help' for more information
    (qemu) 
    (qemu) gdbserver 
    Duplicate ID 'gdb' for chardev
    Could not open gdbserver on device 'tcp::1234'
    (qemu) 

Seems like out of the box, gdb debugging doesn't work? This[1] suggests
otherwise. The documentation here[2] seems to be sparse.


  [1] http://wiki.qemu.org/download/qemu-doc.html#gdb_005fusage
  [2] http://wiki.qemu.org/Documentation/Debugging

-- 
/kashyap

On Thu, Feb 13, 2014 at 12:08:37AM +0530, Kashyap Chamarthy
wrote:
> The 'qemu-kvm' command-line just hung at:
> 
>     $ ./run libguestfs-test-tool
>     [. . .]
>         -chardev
socket,path=/home/tuser1/src/libguestfs/tmp/libguestfspCGc1F/guestfsd.sock,id=channel0
\
>     -device virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
>     -append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check
lpj=2294686 acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0
guestfs_verbose=1 TERM=screen'
It's supposed to "hang" if you add the -S -s arguments.

It is waiting for you to connect gdb to qemu.  See the docs in
src/launch-direct.c.

https://github.com/libguestfs/libguestfs/blob/master/src/launch-direct.c#L404

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

On Thu, Feb 13, 2014 at 09:52:15AM +0000, Richard W.M. Jones
wrote:
> On Thu, Feb 13, 2014 at 12:08:37AM +0530, Kashyap Chamarthy wrote:
> > The 'qemu-kvm' command-line just hung at:
> > 
> >     $ ./run libguestfs-test-tool
> >     [. . .]
> >         -chardev
socket,path=/home/tuser1/src/libguestfs/tmp/libguestfspCGc1F/guestfsd.sock,id=channel0
\
> >     -device
virtserialport,chardev=channel0,name=org.libguestfs.channel.0 \
> >     -append 'panic=1 console=ttyS0 udevtimeout=600 no_timer_check
lpj=2294686 acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0
guestfs_verbose=1 TERM=screen'
> 
> It's supposed to "hang" if you add the -S -s arguments.
Yep! (That indicates how often I use GDB.)

On a more closer look:

    [. . .]
    checking modpath /lib/modules/3.14.0-0.rc2.git0.1.fc21.x86_64 is a directory
    picked kernel vmlinuz-3.14.0-0.rc2.git0.1.fc21.x86_64
    supermin helper [00000ms] finished creating kernel
    [. . .]
    libguestfs: warning: qemu debugging is enabled, connect gdb to tcp::1234 to
begin
    [. . .]
> 
> It is waiting for you to connect gdb to qemu.
Yes:


    (gdb) symbol-file
/usr/lib/debug/lib/modules/3.14.0-0.rc2.git4.1.fc21.x86_64/vmlinux
    Reading symbols from
/usr/lib/debug/lib/modules/3.14.0-0.rc2.git4.1.fc21.x86_64/vmlinux...done.
    (gdb) target remote tcp::1234
    Remote debugging using tcp::1234
    0x0000fff0 in cpu_lock_stats ()
    (gdb) cont
    Continuing.

As I write this, the applinace test (ran on L1) is successful. 
`qemu-sanity-check` doesn't succeed, still investigating.

Thanks.

-- 
/kashyap