Pkg xen devel - Sep 2015 - [Xen-devel] On distro packaging of stub domains (Re: Notes from Xen BoF at Debconf15)

Hi,

Wei Liu hinted that I should "chime in and / or provide corrections" 
(his words).  I'll attempt to do exactly that by not really replying to 
anything specific.  For the record, when I say "we" in this mail, I
mean
"people who have contributed to the rump kernel project" (as also 
indicated by the email-hat).

First of all, there's a difference between a rump kernel (driver bundle 
built out of unmodified kernel components) and any unikernel you 
construct out of rump kernels ... sort of like how there's a difference 
between Linux and GNU/Linux.

For unikernels, the rump kernel project provides Rumprun, which can 
provide you with a near-full POSIX'y interface.  Rumprun also provides 
toolchain wrappers so that you can compile existing programs as Rumprun 
unikernels.  Rumprun also recently regrew the ability to run without the 
POSIX'y bits; some people found it important to be able to make a 
tradeoff between running POSIX'y applications and more compact "kernel 
plane" unikernels such as routers and firewalls.  But, for brevity and 
simplicity, I'll assume the POSIX'y mode for the rest of this email, 
since that's what the QEMU stubdom will no doubt use.

If the above didn't explain the grand scheme of things clearly, have a 
look at http://wiki.rumpkernel.org/Repo and especially the picture.  If 
things are still not clear after that, please point out matters of 
confusion and I will try to improve the explanations.

Also for simplicity, I'll be talking about rump kernels constructed from 
the NetBSD kernel, and the userspace environment of Rumprun being 
NetBSD-derived.  Conceptually, there's nothing stopping someone from 
plugging a GNU layer on top of NetBSD-derived rump kernels (a bit like 
Debian kXBSD?) or constructing rump kernels out of Linux.  But for now, 
let's talk about the only working implementation.

As far as I know, the API/ABI of the application environment provided by 
Rumprun is the same as the one provided by standard NetBSD.  Granted, I 
didn't perform the necessary experiments to verify that, so take the 
following with a few pinches of salt.  In theory, you could take 
application objects built for NetBSD and link them against Rumprun libs. 
  However, since a) nobody (else) ships applications as relocatable 
static objects b) Rumprun does not support shared libraries, I don't 
know how helpful the fact of ABI compatibility is.  IMO, adding shared 
library support would be a backwards way to go: increasing runtime 
processing and memory requirements to solve a build problem sounds plain 
weird.  So, I don't think you can leverage anything existing.

We do have most of the Rumprun cross-toolchain figured out at this 
point.  First, we don't ship any backend toolchain(s), but rather bolt 
wrappers and specs on top of any toolchain (*) you provide.  That way we 
don't have to figure out where to get a toolchain which produces binary 
for every target that everyone might want.  Also, it makes bootstrapping 
Rumprun convenient, since you just say "hey give me the components and 
application wrappers for CC=foocc" and off you go.

*) as long as it's gcc-derived, for now (IIRC gcc 4.8 - 5.1 are known to 
work well, older than that at least C++ won't work).  clang doesn't 
support specs files at least AFAIK, so someone would have to figure out 
how to move the contents of the specs into the wrappers, or whatever 
equivalent clang uses.  (patches welcome ;)

The produced wrappers look exactly like a normal cross-toolchain.  The 
tuple is the same as what NetBSD uses, except with rumprun added in the 
middle, so e.g. x86_64-rumprun-netbsd or arm-rumprun-netbsdelf-eabihf. 
That naming scheme means that most GNU-using software compiles nicely 
for Rumprun just by running configure as ./configure 
--host=x86_64-rumprun-netbsd followed by "make".  Sometimes you 
additionally need things like --disable-shared, but all in all 
everything works pretty well.  See 
http://repo.rumpkernel.org/rumprun-packages for a bunch of "case 
studies", not limited to just GNU autotools.

After "make", before launch we have an additional step called
"bake",
which links the specific kernel components onto the binary.  So for 
example, you can make the compiled binary run on Xen or KVM depending on 
which kernel components you bake onto it.  As a crude analogy, it's like 
scp'ing a binary to a Xen or KVM or bare metal system, but since the 
Rumprun unikernel is missing exec, we use the linker to perform "system 
selection".

So for shipping, one option is to ship the binary after "make", but
then
you also need to ship the toolchain.  The other option is to ship the 
baked binary, but then you lose some of your possibilities on how to 
target the binary.  I'm not sure either option is right for all cases.

We're still trying to figure out the exact form and figure of 
bake+launch.  In the original implementation we assumed that at 
launch-time we could cheaply control all of the details of the backend 
(a la xl.conf).  That assumption proved to be bad not only for example 
for embedded systems (which we should've foreseen), but also for cases 
like Amazon EC2 (where creating something launchable and launching 
something are seriously separate steps).  I'm not going to go into 
details in this thread, but just saying that we still have a few things 
to figure out in the full source-to-execution chain.  If someone wants 
to ship something, please please check with the rump kernel community 
first so that we know that you want to depend on some syntax/semantics 
not changing anymore.

I don't really have good solutions for the packaging problem.  Building 
a "full distro" around rump kernels certainly sounds interesting, and 
we're sort of trying to experiment with that with rumprun-packages. 
However, for packages sooner or later we need to assimilate a real 
packaging system which properly manages dependencies, licenses, 
vulnerabilities, etc.  I'm unsure we'd want to assimilate every existing
packaging system -- Rumprun works the same on every platform, after all. 
  Hard to say for now, I hadn't actually considered the case where a 
distro might want to natively ship binary Rumprun images, as opposed to 
just the toolchain and components.

If I were you folks, I'd start by getting qemu out of the door, and 
worry about generalized solutions when someone wants to ship the second 
unikernel (or third or any small N>1).  If you can't sell to distros 
something that solves a problem, it's unlikely you'll be able to sell a 
framework for solving the same one problem (though, granted, it might be 
easier to sell a framework to computer folk -- "nevermind the solution, 
here's abstraction!").

If you haven't tried rumprun-packages, I recommend to pick something 
from there and see how it works.  Doing so might give some more 
perspective on how easy or difficult it would be to package QEMU.

Whoops, ended up being a bit longer than what I hoped for, but with any 
luck at least some new information was communicated.

   - antti

On Tue, 2015-09-08 at 15:03 +0000, Antti Kantee wrote:
> For unikernels, the rump kernel project provides Rumprun, which can 
> provide you with a near-full POSIX'y interface.
I'm not 100% clear: Does rumprun _build_ or _run_ the application? It sound
s like it builds but the name suggests otherwise.
>   Rumprun also provides 
> toolchain wrappers so that you can compile existing programs as Rumprun 
> unikernels.
Do these wrappers make a rump kernel build target look just like any other
ross build target? (I've just got to the end and found my answer, which was
yes. I've left this next section in since I think it's a nice summary of
why it matters that the answer is yes)

e.g. I have aarch64-linux-gnu-{gcc,as,ld,ar,etc} which I can use to build
aarch64 binaries on my x86_64 host, including picking up aarch64 libraries
and headers from the correct arch specific patch.

Do these rumprun-provided wrappers provide x86_64-rumpkernel
-{gcc,as,ld,ar,etc} ?

Appearing as a regular cross-compilation target is, I think, going to be
important to being able to create rumpkernel based versions of distro
packages.

I think that package maintainers ideally won't want to have to include a
bunch of rumpkernel specific code in their package, they just want to
leverage the existing cross-compilability of their package.

This may not be super critical for a standalone package with a tiny build
dependency chain. Consider building a QEMU rumpkernel to be shipped in a
Debian package. The existing QEMU source package in Debian has ~30 library
dependencies, many of which I expect have their own dependencies etc. For
another metric consider for the regular x86 binary:

$ ldd /usr/bin/qemu-system-x86_64  | wc -l
87
$

So without a normal looking cross env we are talking about asking maybe 60
-90 different library maintainers to do something special for rumpkernels.

(Surely some of those 60-80 are things you would omit from a rump build for
various reasons, but still...)
> If the above didn't explain the grand scheme of things clearly, have a 
> look at http://wiki.rumpkernel.org/Repo and especially the picture.  If 
> things are still not clear after that, please point out matters of 
> confusion and I will try to improve the explanations.
I think that wiki page is clear, but I think it's orthogonal to the issue
with distro packaging of rump kernels.
>   However, since a) nobody (else) ships applications as relocatable 
> static objects b) Rumprun does not support shared libraries, I don't 
> know how helpful the fact of ABI compatibility is.  IMO, adding shared 
> library support would be a backwards way to go: increasing runtime 
> processing and memory requirements to solve a build problem sounds plain 
> weird.  So, I don't think you can leverage anything existing.
This is an interesting point, since not building a shared library is
already therefore requiring packaging changes which are going to be at
least a little bit rumpkernel specific.

Is it at all possible (even theoretically) to take a shared library (which
is relocatable as required) and to do a compile time static linking pass on
it? i.e. use libfoo.so but still do static linking?
> We do have most of the Rumprun cross-toolchain figured out at this 
> point.  First, we don't ship any backend toolchain(s), but rather bolt 
> wrappers and specs on top of any toolchain (*) you provide.  That way we 
> don't have to figure out where to get a toolchain which produces binary
> for every target that everyone might want.  Also, it makes bootstrapping 
> Rumprun convenient, since you just say "hey give me the components and
> application wrappers for CC=foocc" and off you go.
> The produced wrappers look exactly like a normal cross-toolchain.  The 
> tuple is the same as what NetBSD uses, except with rumprun added in the 
> middle, so e.g. x86_64-rumprun-netbsd or arm-rumprun-netbsdelf-eabihf. 
Great, I think this solves a large problem (whether its a wrapper or a
"proper cross compiler" I think is of limited importance as long as it
behaves enough like the latter)
> I don't really have good solutions for the packaging problem.  Building
> a "full distro" around rump kernels certainly sounds interesting,
FWIW I don't think we need a full distro, just sufficient build
dependencies for the actual useful things (maybe that converges on to a
full distro though).
>   Hard to say for now, I hadn't actually considered the case where a 
> distro might want to natively ship binary Rumprun images, as opposed to 
> just the toolchain and components.
One concrete example of what I want is a Debian package in the Debian
archive which the xen-tools.deb can depend on which provides a qemu
rumpkernel binary such that users can use stubdomain functionality for
their HVM guests.

Debian are (most likely) not going to accept a second copy of the QEMU
source in the archive and likewise they wouldn't want a big source package
which was "qemu + all its build dependencies" or anything like that,
especially when "all its build dependencies" is duplicating the source
of
dozens of libraries already in Debian.
> If I were you folks, I'd start by getting qemu out of the door
That's certainly the highest priority, at the moment I don't think we
actually have a QEMU Xen dm based on rumpkernels which anyone could package
anyway, irrespective of how hard that might be.
> , and 
> worry about generalized solutions when someone wants to ship the second 
> unikernel (or third or any small N>1).
Unfortunately I think the N==1 case is tricky already from a distro
acceptance PoV. (At least for Binary distros, it's probably trivial in a
Source based distro like Gentoo)

Ian.

Ian Campbell, le Tue 08 Sep 2015 17:15:40 +0100, a ?crit
:
> Is it at all possible (even theoretically) to take a shared library (which
> is relocatable as required) and to do a compile time static linking pass on
> it? i.e. use libfoo.so but still do static linking?
? gcc test.c -o libtest.so -shared -Wl,--relocatable
/usr/bin/ld.bfd.real: -r and -shared may not be used together

Samuel

On 08/09/15 16:15, Ian Campbell wrote:
> On Tue, 2015-09-08 at 15:03 +0000, Antti Kantee wrote:
>
>> For unikernels, the rump kernel project provides Rumprun, which can
>> provide you with a near-full POSIX'y interface.
>
> I'm not 100% clear: Does rumprun _build_ or _run_ the application? It
sound
> s like it builds but the name suggests otherwise.
For all practical purposes, Rumprun is an OS, except that you always 
cross-compile for it.  So, I'd say "yes", but it depends on how
you want
to interpret the situation.  We could spend days writing emails back and 
forth, but there's really no substitute for an hour of hands-on 
experimentation.

(nb. the launch tool for launching Rumprun instances is currently called 
rumprun.  It's on my todo list to propose changing the name of the tool 
to e.g. rumprunner or runrump or something which is distinct from the OS 
name, since similarity causes some confusion)
> Do these wrappers make a rump kernel build target look just like any other
> ross build target? (I've just got to the end and found my answer, which
was
> yes. I've left this next section in since I think it's a nice
summary of
> why it matters that the answer is yes)
>
> e.g. I have aarch64-linux-gnu-{gcc,as,ld,ar,etc} which I can use to build
> aarch64 binaries on my x86_64 host, including picking up aarch64 libraries
> and headers from the correct arch specific patch.
>
> Do these rumprun-provided wrappers provide x86_64-rumpkernel
> -{gcc,as,ld,ar,etc} ?
No, like I said and which you discovered later, 
x86_64-rumprun-netbsd-{gcc,as,ld,ar,etc}.  aarch64 would be 
aarch64-rumprun-netbsd-{...}.
> Appearing as a regular cross-compilation target is, I think, going to be
> important to being able to create rumpkernel based versions of distro
> packages.
>
> I think that package maintainers ideally won't want to have to include
a
> bunch of rumpkernel specific code in their package, they just want to
> leverage the existing cross-compilability of their package.
Yes, that is critical.  We bled to achieve that goal.  It looks obvious 
now, but I can assure you it wasn't obvious a year ago.
> $ ldd /usr/bin/qemu-system-x86_64  | wc -l
> 87
> $
Heh, that's quite a lot.
>> If the above didn't explain the grand scheme of things clearly,
have a
>> look at http://wiki.rumpkernel.org/Repo and especially the picture.  If
>> things are still not clear after that, please point out matters of
>> confusion and I will try to improve the explanations.
>
> I think that wiki page is clear, but I think it's orthogonal to the
issue
> with distro packaging of rump kernels.
Sure, but I wanted to get the concepts right.  And they're still not 
right.  We're talking about packaging for *Rumprun*, not rump kernels in 
general.
>>    However, since a) nobody (else) ships applications as relocatable
>> static objects b) Rumprun does not support shared libraries, I
don't
>> know how helpful the fact of ABI compatibility is.  IMO, adding shared
>> library support would be a backwards way to go: increasing runtime
>> processing and memory requirements to solve a build problem sounds
plain
>> weird.  So, I don't think you can leverage anything existing.
>
> This is an interesting point, since not building a shared library is
> already therefore requiring packaging changes which are going to be at
> least a little bit rumpkernel specific.
>
> Is it at all possible (even theoretically) to take a shared library (which
> is relocatable as required) and to do a compile time static linking pass on
> it? i.e. use libfoo.so but still do static linking?
But shared libraries aren't "relocatable", that's the whole
point of
shared libraries! ;) ;)

I guess you could theoretically link shared libs with a different ld, 
and I don't think it would be very different from prelinking shared 
libs, but as Samuel demonstrated, it won't work at least with an 
out-of-the-box ld.

I think it's easier to blame Solaris for the world going bonkers with 
shared libs, bite the bullet, and start adding static linking back where 
it's been ripped out from.  Shared libs make zero sense for unikernels 
since you don't have anyone to share them with, so you're just paying 
extra for PIC for absolutely no return.  (dynamically loadable code is a 
separate issue, if you even want to go there ... I wouldn't)
>> I don't really have good solutions for the packaging problem. 
Building
>> a "full distro" around rump kernels certainly sounds
interesting,
>
> FWIW I don't think we need a full distro, just sufficient build
> dependencies for the actual useful things (maybe that converges on to a
> full distro though).
By "full distro" I meant "enough to get a majority of the useful 
services going".  Seems like once qemu works, we're 99% there ;)
> Debian are (most likely) not going to accept a second copy of the QEMU
> source in the archive and likewise they wouldn't want a big source
package
> which was "qemu + all its build dependencies" or anything like
that,
> especially when "all its build dependencies" is duplicating the
source of
> dozens of libraries already in Debian.
Why do you need a second copy of the sources?  Or are sources always 
strictly associated with one package without any chances of pulling from 
a master package?  You are going to need two copies of the binaries 
anyway, so it doesn't seem like a particularly big deal to me, not that 
I'm questioning your statement.
>> If I were you folks, I'd start by getting qemu out of the door
>
> That's certainly the highest priority, at the moment I don't think
we
> actually have a QEMU Xen dm based on rumpkernels which anyone could package
> anyway, irrespective of how hard that might be.
>
>> , and
>> worry about generalized solutions when someone wants to ship the second
>> unikernel (or third or any small N>1).
>
> Unfortunately I think the N==1 case is tricky already from a distro
> acceptance PoV. (At least for Binary distros, it's probably trivial in
a
> Source based distro like Gentoo)
Ok.  I'll help where I can, but I don't think I can be the primus motor 
for solving the distro acceptance problem for Xen stubdomains.

If you can say to the packaging system "build with this cross toolchain 
but disable shared" you're already quite far along, and it seems like 
something that shouldn't be too difficult to get reasonable packaging 
systems to support.  But, details, details.  One major detail is that 
your target is quite wide, and not everyone along that target can be 
assumed to be reasonable :/