Libguestfs - Apr 2014 - Adventures in building libguestfs on non-x86 architectures for Debian

Hi,

things are progressing slowly, but I feel that eventually I'll be
getting there: supermin can be built on all Linux-based architectures
that are part of Debian/unstable[1] -- (except for sparc, but I'll
ignore that for now.) libguestfs on the other hand currently fails
launching the appliance (necessary for running tests) on everything but
x86 and mips.

Apparently, qemu-system-arm 2.0 as currently available through
Debian/unstable doesn't want to tell us anything about available devices
unless we specify a machine type. The patch below works around that
issue, but I haven't been able to run qemu-system-arm with a supermin
appliance on Debian's armhf porterbox so far: 100% CPU usage, no output
from the kernel.

For armel, there's no kernel with device tree blobs, so I think that
I'll
need to use linux-image-versatile there or ignore armel altogether.

powerpc: qemu-system-ppc does not know about the "pseries" machine, I
have figured out that I need to use qemu-system-ppc64 for that. So far,
I haven't been able to manually boot a Supermin appliance using
qemu-system-ppc, either, though.

s390x does not seem to know about virtio-blk-pci.

Cheers,
-Hilko

[1] https://buildd.debian.org/status/package.php?p=supermin
[2] https://buildd.debian.org/status/package.php?p=libguestfs

diff --git a/src/launch-direct.c b/src/launch-direct.c
index 1460c56..2332368 100644
--- a/src/launch-direct.c
+++ b/src/launch-direct.c
@@ -1020,6 +1020,10 @@ test_qemu (guestfs_h *g, struct backend_direct_data
*data)
   guestfs___cmd_add_arg (cmd3, g->hv);
   guestfs___cmd_add_arg (cmd3, "-display");
   guestfs___cmd_add_arg (cmd3, "none");
+#ifdef MACHINE_TYPE
+  guestfs___cmd_add_arg (cmd3, "-M");
+  guestfs___cmd_add_arg (cmd3, MACHINE_TYPE);
+#endif
   guestfs___cmd_add_arg (cmd3, "-machine");
   guestfs___cmd_add_arg (cmd3, "accel=kvm:tcg");
   guestfs___cmd_add_arg (cmd3, "-device");

On Tue, Apr 29, 2014 at 11:18:01PM +0200, Hilko Bengen
wrote:
> Hi,
> 
> things are progressing slowly, but I feel that eventually I'll be
> getting there: supermin can be built on all Linux-based architectures
> that are part of Debian/unstable[1] -- (except for sparc, but I'll
> ignore that for now.) libguestfs on the other hand currently fails
> launching the appliance (necessary for running tests) on everything but
> x86 and mips.
> 
> Apparently, qemu-system-arm 2.0 as currently available through
> Debian/unstable doesn't want to tell us anything about available
devices
> unless we specify a machine type. The patch below works around that
> issue, but I haven't been able to run qemu-system-arm with a supermin
> appliance on Debian's armhf porterbox so far: 100% CPU usage, no output
> from the kernel.
ARM is always fun to debug, but 32 bit ARM should be working.  It
works for me anyway (using Fedora and qemu 1.x).

The way to debug your spin is to use gdb (the following assumes you
have access to the hardware; if you don't, my recommendation is to buy
a Cubietruck!).  Uncomment the following lines of code:

https://github.com/libguestfs/libguestfs/blob/master/src/launch-direct.c#L406

When libguestfs runs, it will pause when the appliance starts.  Attach
gdb to the appliance by following the instructions there, and 'cont'
it, then when it spins hit ^C and get a stack trace to see where in
the kernel it got stuck.  You can also dump the kernel log (log_buf)
to see if there are any lost kernel messages.

We should have aarch64 working in a matter of weeks.

The problem is keeping it working, since we have no CI-style tests on
these architectures.  Anything from kernel updates to qemu updates to
libguestfs changes can break it, and we wouldn't notice for a while.
> For armel, there's no kernel with device tree blobs, so I think that
I'll
> need to use linux-image-versatile there or ignore armel altogether.
> 
> powerpc: qemu-system-ppc does not know about the "pseries"
machine, I
> have figured out that I need to use qemu-system-ppc64 for that. So far,
> I haven't been able to manually boot a Supermin appliance using
> qemu-system-ppc, either, though.
ppc64 ought to work.  At least, it WFM (on Fedora ...).

ppc64le is a different matter.  So far I've not been able to get
access to any POWER8 hardware.  However Michel Normand from IBM has
looked at getting the virt stack working on ppc64le on Fedora.

I haven't tried ppc 32 bit for a long time, and don't particularly
care about it.  AFAIK the hardware is obsolete.
> s390x does not seem to know about virtio-blk-pci.
s390x is problematic in a few ways:

 - No OCaml native compiler

 - KVM exists but is completely unknown territory.

I have access to real s390x hardware but I've not spent any serious
time trying to get libguestfs to work.
> [1] https://buildd.debian.org/status/package.php?p=supermin
> [2] https://buildd.debian.org/status/package.php?p=libguestfs
> 
> diff --git a/src/launch-direct.c b/src/launch-direct.c
> index 1460c56..2332368 100644
> --- a/src/launch-direct.c
> +++ b/src/launch-direct.c
> @@ -1020,6 +1020,10 @@ test_qemu (guestfs_h *g, struct backend_direct_data
*data)
>    guestfs___cmd_add_arg (cmd3, g->hv);
>    guestfs___cmd_add_arg (cmd3, "-display");
>    guestfs___cmd_add_arg (cmd3, "none");
> +#ifdef MACHINE_TYPE
> +  guestfs___cmd_add_arg (cmd3, "-M");
> +  guestfs___cmd_add_arg (cmd3, MACHINE_TYPE);
> +#endif
>    guestfs___cmd_add_arg (cmd3, "-machine");
>    guestfs___cmd_add_arg (cmd3, "accel=kvm:tcg");
>    guestfs___cmd_add_arg (cmd3, "-device");
ACK.

Thanks for looking into this!

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

* Richard W.M. Jones:
>> Apparently, qemu-system-arm 2.0 as currently available through
>> Debian/unstable doesn't want to tell us anything about available
devices
>> unless we specify a machine type. The patch below works around that
>> issue, but I haven't been able to run qemu-system-arm with a
supermin
>> appliance on Debian's armhf porterbox so far: 100% CPU usage, no
output
>> from the kernel.
>
> ARM is always fun to debug, but 32 bit ARM should be working.  It
> works for me anyway (using Fedora and qemu 1.x).
I just got around to revisiting the problem. Using
qemu-system-arm_2.0.0+dfsg-6 from Debian/unstable, booting vexpress-a15
doesn't seem to work, but vexpress-a9 does. I successfully built and did
a "make quickcheck" on an armhf porterbox with the patch attached
below.
Is there any specific reason why -a15 was chosen that might make this a
bad idea?

I alos noticed that there seems to be a plan to use -M virt at some
point for 32bit arm. Can qemu generate the device tree internally yet?

Cheers,
-Hilko