Libguestfs - Apr 2013 - launch failure of appliance with nbd drive

I'm using qemu-nbd to setup the device with an offset
into the image. Using libguestfs version 1.21.22

Code for adding the drive:
  char *server[] = { "localhost:3000", NULL };
  if (guestfs_add_drive_opts (g, "" /* export name - see below */,
     GUESTFS_ADD_DRIVE_OPTS_FORMAT, "raw",
     GUESTFS_ADD_DRIVE_OPTS_PROTOCOL, "nbd",
     GUESTFS_ADD_DRIVE_OPTS_SERVER, server,
     -1)

Here is some of the stdout. A mix of libguestfs and
qemu-nbd messages.

	qemu-nbd -v -n -o 77824 -t -p 3000 -c /dev/nbd0
/home/skippy/mnt/mnt0/win7x64amdHP.vdi
	NBD device /dev/nbd0 is now connected to /home/skippy/mnt/mnt0/win7x64amdHP.vdi

	libguestfs: launch: attach-method=appliance
	libguestfs: launch: tmpdir=/tmp/libguestfsLFMzZI
	libguestfs: launch: umask=0022
	libguestfs: launch: euid=0
	libguestfs: [00000ms] begin testing qemu features
	libguestfs: command: run: /usr/bin/qemu-system-x86_64
	libguestfs: command: run: \ -nographic
	libguestfs: command: run: \ -help
	libguestfs: command: run: /usr/bin/qemu-system-x86_64
	libguestfs: command: run: \ -nographic
	libguestfs: command: run: \ -version
	libguestfs: qemu version 1.2
	libguestfs: command: run: /usr/bin/qemu-system-x86_64
	libguestfs: command: run: \ -nographic
	libguestfs: command: run: \ -machine accel=kvm:tcg
	libguestfs: command: run: \ -device ?
	libguestfs: [00131ms] finished testing qemu features

	\ libguestfs: child_cleanup: 0xbc4ba0: child process died
	libguestfs: sending SIGTERM to process 23981

	libguestfs: closing guestfs handle 0xbc4ba0 (state 0)
	libguestfs: command: run: rm
	libguestfs: command: run: \ -rf /tmp/libguestfsLFMzZI
	skippy at DaneelOlivaw:~/tmp/bin$
/build/buildd/qemu-kvm-1.2.0+noroms/nbd.c:nbd_trip():L835: From:
18446744073709551104, Len: 0, Size: 20003880960, Offset: 77824

	/build/buildd/qemu-kvm-1.2.0+noroms/nbd.c:nbd_trip():L836: requested
operation past EOF--bad client?
	/build/buildd/qemu-kvm-1.2.0+noroms/nbd.c:nbd_receive_request():L516:
read failed

So I peppered some fprintfs in launch-appliance.c to try and narrow
down the problem. Just below the comment:
   * does not mean the daemon is up until we read the
   * GUESTFS_LAUNCH_FLAG below.  Failures in qemu startup can still
Is the call where it fails:
  r = guestfs___recv_from_daemon (g, &size, &buf);
r is -1 and size is zero

How can I debug this problem further?
I suppose I could build/debug the qemu side of this process...

-Skippy

On Thu, Apr 04, 2013 at 10:48:54AM -0400, Skippy VonDrake
wrote:
> I'm using qemu-nbd to setup the device with an offset
> into the image. Using libguestfs version 1.21.22
>
> Code for adding the drive:
>   char *server[] = { "localhost:3000", NULL };
>   if (guestfs_add_drive_opts (g, "" /* export name - see below
*/,
>      GUESTFS_ADD_DRIVE_OPTS_FORMAT, "raw",
>      GUESTFS_ADD_DRIVE_OPTS_PROTOCOL, "nbd",
>      GUESTFS_ADD_DRIVE_OPTS_SERVER, server,
>      -1)
Glad someone is trying it!
> Here is some of the stdout. A mix of libguestfs and
> qemu-nbd messages.
> 
> 	\ libguestfs: child_cleanup: 0xbc4ba0: child process died
> 	libguestfs: sending SIGTERM to process 23981
[...]

Are you sure that qemu itself isn't crashing?

I've only tested this on qemu 1.4.0.
> 	/build/buildd/qemu-kvm-1.2.0+noroms/nbd.c:nbd_trip():L836: requested
> operation past EOF--bad client?
> 	/build/buildd/qemu-kvm-1.2.0+noroms/nbd.c:nbd_receive_request():L516:
> read failed
> 
> So I peppered some fprintfs in launch-appliance.c to try and narrow
> down the problem. Just below the comment:
>    * does not mean the daemon is up until we read the
>    * GUESTFS_LAUNCH_FLAG below.  Failures in qemu startup can still
> Is the call where it fails:
>   r = guestfs___recv_from_daemon (g, &size, &buf);
> r is -1 and size is zero
This would indicate that qemu is exiting or crashing early.
> How can I debug this problem further?
> I suppose I could build/debug the qemu side of this process...
If qemu is crashing then enabling core dumps might be a good idea, ie:

  ulimit -c unlimited

But even if you get a coredump, no one in qemu-land will care since
1.2.0 is an old version.  You could try a locally compiled copy of
qemu from git:

  http://libguestfs.org/guestfs.3.html#qemu-wrappers

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

You may be interested that libguestfs 1.21.30 has support for
remote URIs from guestfish etc.

  guestfish -a ssh://remote/path/to/disk.img

In upstream git also Unix domain sockets are supported:

  guestfish -a nbd://?socket=/tmp/sock

More here:
https://rwmj.wordpress.com/2013/04/16/use-guestfish-virt-tools-with-remote-disks/

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v