Libguestfs - Apr 2013 - [Qemu-devel] debugging qemu-nbd.c

On Tue, Apr 09, 2013 at 10:18:50AM -0400, Skippy VonDrake
wrote:
> Hello,
> 
> Apologies upfront if posting to this list is inappropriate.
> 
> I have an app that forks a process with calls qemu-nbd with an offset
> into an image.
> The app then uses libguestfs to attach the nbd device.
> But it appears that qemu is failing.
When you say "qemu is failing", do you mean qemu-nbd, or the qemu
subprocess that libguestfs uses?  Does it print any error?
> I'm using the latest qemu sources and placed file log statements within
> qemu-nbd.c to pinpoint the problem.
> 
> The statement: " if (chdir("/") < 0)"
> appears to be the culprit.
> 
> But the subsequent err() call is never reached. If I pull 'chdir'
out of
> the 'IF' and assign a local var to its return - execution still
seems to
> stop on the 'chdir'. So there is no opportunity to examine its
return.
> 
> What is the best approach to debugging this section of code?
As a first step I would just add 'fprintf (stderr, ...)' straight into
the qemu-nbd.c file.  Or run it under gdb.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

On Tue, Apr 09, 2013 at 10:41:47AM -0400, Skippy VonDrake
wrote:
> On Tue, Apr 9, 2013 at 10:33 AM, Richard W.M. Jones <rjones at
redhat.com> wrote:
> > On Tue, Apr 09, 2013 at 10:18:50AM -0400, Skippy VonDrake wrote:
> >> Hello,
> >>
> >> Apologies upfront if posting to this list is inappropriate.
> >>
> >> I have an app that forks a process with calls qemu-nbd with an
offset
> >> into an image.
> >> The app then uses libguestfs to attach the nbd device.
> >> But it appears that qemu is failing.
> >
> > When you say "qemu is failing", do you mean qemu-nbd, or the
qemu
> > subprocess that libguestfs uses?  Does it print any error?
> 
> I believe it is the qemu subprocess that libguestfs is using.
> I'm basing that assumption on the log file statements from both
libguestfs
> and qemu, and from stderr from libguestfs.
> 
> The only stderr from libguestfs is "child_cleanup:... child process
died"
Enable debugging in libguestfs to find out what's going on:

http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
> >> I'm using the latest qemu sources and placed file log
statements within
> >> qemu-nbd.c to pinpoint the problem.
> >>
> >> The statement: " if (chdir("/") < 0)"
> >> appears to be the culprit.
> >>
> >> But the subsequent err() call is never reached. If I pull
'chdir' out of
> >> the 'IF' and assign a local var to its return - execution
still seems to
> >> stop on the 'chdir'. So there is no opportunity to examine
its return.
> >>
> >> What is the best approach to debugging this section of code?
> >
> > As a first step I would just add 'fprintf (stderr, ...)'
straight into
> > the qemu-nbd.c file.  Or run it under gdb.
> 
> I haven't tried this with gdb yet but the log statements I described
> above were in the qemu-nbd.c file. And are successfully written
> up until the chdir command.
> 
> I'll try attaching gdb...
Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)

---------- Forwarded message ----------
From: Skippy VonDrake <skippyvondrake at gmail.com>
Date: Tue, Apr 9, 2013 at 11:38 AM
Subject: Re: [Qemu-devel] debugging qemu-nbd.c
To: "Richard W.M. Jones" <rjones at redhat.com>

>> >> I have an app that forks a process with calls qemu-nbd with an
offset
>> >> into an image.
>> >> The app then uses libguestfs to attach the nbd device.
>> >> But it appears that qemu is failing.
>> >
>> > When you say "qemu is failing", do you mean qemu-nbd, or
the qemu
>> > subprocess that libguestfs uses?  Does it print any error?
>>
>> I believe it is the qemu subprocess that libguestfs is using.
>> I'm basing that assumption on the log file statements from both
libguestfs
>> and qemu, and from stderr from libguestfs.
>>
>> The only stderr from libguestfs is "child_cleanup:... child
process died"
>
> Enable debugging in libguestfs to find out what's going on:
>
> http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
With both LIBGUESTFS_DEBUG and LIBGUESTFS_TRACE set and
guestfs_set_verbose(g,1) and guestfs_set_trace(g,1).
Below is the output.
Some messages may look unfamiliar because I placed some
statements in libguestfs just for a 'crude' tracing into the library.
The initial 'nbd_client_thread' message is one I placed in the qemu
sources.

executing 'system(sudo rmmod nbd)
executing 'system(sudo modprobe nbd max_part=16)
setupNbdServer - size: 77824
setupNbdServer - system(serverCmd) string: qemu-nbd -v -n -o 77824 -t
-p 3000 -c /dev/nbd0 /home/skippy/mnt/mnt0/win7x64amdHP.vdi
nbd_client_thread:line 260::12806|939447:: qemu-NBD device /dev/nbd0
is NOWWW connected to /home/skippy/mnt/mnt0/win7x64amdHP.vdi
libguestfs: trace: add_drive "" "format:raw"
"protocol:nbd"
"server:localhost:3000"
libguestfs: trace: add_drive = 0
guestfs is launching the appliance
libguestfs: trace: launch
libguestfs: trace: get_tmpdir
libguestfs: trace: get_tmpdir = "/tmp"
libguestfs: launch: attach-method=appliance
libguestfs: launch: tmpdir=/tmp/libguestfsscRysl
libguestfs: launch: umask=0022
libguestfs: launch: euid=0
libguestfs: [00000ms] begin testing qemu features
libguestfs: command: run:
/home/skippy/Downloads/qemu/latest-build/install_dir/bin/qemu-system-x86_64
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -help
libguestfs: command: run:
/home/skippy/Downloads/qemu/latest-build/install_dir/bin/qemu-system-x86_64
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -version
libguestfs: qemu version 1.4
libguestfs: command: run:
/home/skippy/Downloads/qemu/latest-build/install_dir/bin/qemu-system-x86_64
libguestfs: command: run: \ -nographic
libguestfs: command: run: \ -machine accel=kvm:tcg
libguestfs: command: run: \ -device ?
libguestfs: [00025ms] finished testing qemu features
libguestfs: [00025ms] r from fork: 13658 \n
 \ libguestfs: error: appliance closed the connection unexpectedly,
see earlier error messages
libguestfs: child_cleanup: 0x11f4ba0: child process died
libguestfs: sending SIGTERM to process 13658
libguestfs: error:
/home/skippy/Downloads/qemu/latest-build/install_dir/bin/qemu-system-x86_64
exited with error status 1, see debug messages above
libguestfs: error: guestfs_launch failed, see earlier error messages
libguestfs: trace: launch = -1 (error)
ERROR from setupGuestfs-> guestfs_launch, errnum: 0
Last error msg: guestfs_launch failed, see earlier error messages
main thread exiting
libguestfs: trace: close
libguestfs: closing guestfs handle 0x11f4ba0 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfsscRysl

As you can see, the output looks similar to my previous post. At which you
suggested that qemu might be failing. So that is when I started trying to
pinpoint the problem in qemu. And the 'chdir' command was the final
point
at which my file-log statements ended. For that logging I'd open the log,
write a string and close the log each time.

One bit that might be of interest - my app does a fork/system command
to launch the ndb server. When I give the disconnect command (after the
error output above), I see this displayed from qemu:
  $ nbd.c:nbd_trip():L1106: From: 18446744073709551104, Len: 0, Size:
20003880960, Offset: 77824
  nbd.c:nbd_trip():L1107: requested operation past EOF--bad client?

The image is 20GB with a VDI header prepended to it. The size of
the header is displayed with "setupNbdServer - size: 77824" and
used for the offset parameter for qemu-nbd.

I don't know what to make of the "From" value nor the
'Len' of zero
in the last output.