Pkg xen devel - Sep 2015 - Notes from Xen BoF at Debconf15

Xen upstream BoF
===============
We had a discussion around Xen and packaging at Debian's annual developer
conference (Debconf) a few weeks back:
https://summit.debconf.org/debconf15/meeting/279/xen-upstream-bof/

These are my notes, I think there is probably stuff of interest to most
distro people, not just Debian folks.

The session was scheduled in a small, out of the way, room. Around 2
dozen people attended including:

  Ian Jackson
  Bastian Blank ("waldi", current Xen package maintainer in Debian)
  Guido Trotter (previous Xen package maintainer, who still takes care
                 of stable security updates)
  Axel Beckert  (maintainer of the xen-tools helpers)
  Various users
  Myself

The majority of the conversation was between Ian & I and Bastian and
Guido, some users raised some issues towards the end.

Embedding in xen.git
===================
We are much better about providing ways to use system-supplied
components these days (since 4.4) and Debian uses them.

Waldi noted that iPXE did not have such an option. Since that iPXE is
only used by qemu-trad (for qemu-upstream the iPXE comes from the PCI
ROM BAR) and Debian disabled qemu-trad it should be pretty quick to
patch the build system to disable iPXE.

Secondly it was noted that SeaBIOS is still built into hvmloader,
which makes package updates harder (needs a binNMU of the Xen package
to pickup a new SeaBIOS). Since this is in guest context it is not an
issue for the security team (which would make it higher priority) and
there are medium term plans to perhaps make it possible to load the
BIOS via the toolstack instead.

Note that OVMF would also be built in but is non-free and hence
disabled.

Lowlevel library API stability
=============================
The majority of the current Debian patch queue is moving unstable
libraries (mainly libxc) from $libdir to $libexecdir/xen-X.Y, adding
-rpath where needed and removing the SONAME from such libraries. Also
move related binary files.

Waldi expressed hope that the hypervisor interfaces could become
stable, but we think this is unlikely. Having the hypervisor provide a
compat layer for older interfaces was ruled out as the wrong place
from a security PoV. Second choice would be to have the tools provide
a compat layer for older hypervisors, which would be possible but
perhaps tricky to achieve.

This is also a problem for some third packages, e.g. qemu-upstream and
kexec-tools. This requires a binNMU of those packages to build against
a new Xen package. This is painful for maintainers.

We explained our plan was to move some sections of the unstable
library out into small stable libraries for specific purposes, with
stuff needed for qemu-upstream, kexec-tools and other external
packages being a priority in the short term. After this we plan to
reexamine what is left and consider next steps.

In the meantime it should be much easier these days to provide
upstream configure options to provide the changes currently patched in
by Debian.

Midlevel library stability
=========================
libxenlight is only API not ABI stable. This is a pain in particular
for libvirt which needs binNMU for new Xen package.

We would like to eventually offer ABI stability or this library, but
we are not there yet.

Stubdomains
==========
Hard to do in a packaging environment (is really its own partial
architecture). Rump kernels are no different in this regard.

No clever ideas were put forward.

initscripts
==========
Debian has its own initscripts, does not use the upstream ones.

Waldi stated this was because the upstream ones were not properly LSB
and were too "cross-distro".

We would like to try and have these in xen.git. Perhaps a Yakk, but
closer to upstream the better. Not a very high priority though.

grub-xen
=======
Needs much better docs.

ACTION: I agreed to move the text of my blog post somewhere more
obvious.

Release cycle
============
Waldi commented that the stable release cycle was too long. Would like
to see a release after any large security update.

We asked if the RCs for stable releases were valuable, the answer was
"not so much".

Waldi would prefer to avoid cherry-picking security fixes if possible.

We asked if we thought Xen stable releases could be added to Debian
point releases. Waldi thought they likely could be, citing the
inclusion of Linux stable releases in point releases.

Our stable releases follow a similar set of rules to Linux, we think
we implement them more faithfully (less feature or feature-like
backports)

ACTION: Talk to Jan about making changes to stable release process.

Security updates
===============
Guido asked if security updates could go back further.

Currently we go to 4.2, but Debian Wheezy has Xen 4.1.

The security team don't currently have the effort to go further, but
have recently introduced a private discussion list where predisclosure
members are encouraged to exchange their own backports.

Guido is not on global team at security.debian. We suggested he discuss
with the Debian security team switching to a xen specific alias
including team@ + relevant package maintainers.

Release schedule vs. migration N=>N+1 support
============================================
Philip Hahn (a user) asked what happens to migration if the release
cycle shortens.

We answered that this N=>N+1 policy would need rethinking into an N=>N+M.

We agreed that it would be useful if M was > Debian release cycle (~2
years).

Recent rewrite of migration support has made changing this policy far
more plausible.

It was suggested as an aside that using -backports more would be
useful.

Remus
====
A user (Kai ???) was interested in Remus support.

We briefly discussed status, we think it should be reasonably easy to
combine xl remus into one of the HA systems in Debian (e.g. Pacemaker,
linux-ha?)

Ferenc Wagner pointed out that integrating VMs (non-HA style) into a
pacemake system was quite easy.

Testing Xen in KVM
=================
A user (anon) asked if we tested this, because it sometimes broke.

We only test Xen in Xen, pointed to the wiki page.

OpenStack/xapi in Debian
=======================
I spoke to a couple of users through the week who were asking after
the future of xapi in Debian, because they wanted OpenStack.

I was able to point them to the libvirt+XenProject stuff and explained
about the CI loop etc and they went away happy.

On Tue, 2015-09-08 at 10:24 +0100, Ian Campbell wrote:
> Stubdomains
When I passed these notes around internally for sanity checking we ended up
discussing this issue, since we decided it would be better to move this to
xen-devel I'm quoting the thread below with permission.

It was a bit tricky to flatten the thread into a single mail, but it wasn't
too "branchy" so I think this will be ok. Hopefully I've not
misrepresented
what anyone said by how I've arranged it here...
> ==========> 
> Hard to do in a packaging environment (is really its own partial
> architecture). Rump kernels are no different in this regard.
>
> No clever ideas were put forward.
-----------------------------------------------------------------------

George Dunplap:
> Meaning, hard to reuse binaries from existing qemu package rather than
> using the Xen build system to download and build bespoke qemu images.
> 
> Are the grub packagers doing anything with grub-xen?
> 
> If people want to use the existing qemu package, it does seem like
> making a qemu-xen-stubdom package would be the most sensible thing to do.
-----------------------------------------------------------------------

Me in reply to George:
> From the PoV of a distro a qemu-xen-stubdom package is duplicating a load
> of source code (e.g. qemu and all the libraries which it uses, as well as
> the stubdom libc and kernel etc)), which is disliked by distros, who wish
> to use their existing packages for things.
> 
> But you can't just use the existing source packages for all those
libraries
> rebuilt for "arch=stubdom" since "stubdom" is not an
architecture which the
> distro understands. (An arch to a distro is a processor architecture + libc
> + calling convention ABI). arch=stubdom could never be a full arch to the
> distro (hence "partial architecture").
> 
> Security teams also don't like things which contain duplicates of
source
> code.
> 
> grub-xen is completely self contained, it doesn't rely on anything
outside
> of grub.git(or .tar). That's not because they've imported a load of
3rd
> party libraries, it really is just that grub does PV by itself in a
> completely self contained way, but grub is simple enough to get away with
> this.
-----------------------------------------------------------------------

In a short diversion on the final ("grub-xen is completely...") para
George
said:
> Do you mean that it's easier to just toss grub-xen into the grub
> package, because it's not very big and doesn't require the
maintainer to
> know anything at all about Xen (hence the point about importing 3rd
> party libraries)?
To which I replied:
> Size is not the issue. The fact that grub-xen (or indeed grub-*) has no
> build dependencies at all is what matters.
-----------------------------------------------------------------------

The main thread of the conversation (actually the same mails as above) was in
reply to the "From the PoV of a distro" where George said:
> Sorry, this comment came from an RPM perspective, where a single .spec
> file will build a suite of separate rpms (with the same prefix).  The
> CentOS xen.spec will build xen-[version], xen-hypervisor-[version],
> xen-runtime-[version], xen-libs-[version], xen-ocaml-[version], &c.
> 
> CentOS is a bit of a special case, but I presume that in Fedora, if they
> wanted to, they could build a qemu-xenpv package as part of the normal
> qemu RPM build process that would generate a runnable PV image as part
> of the normal rpmbuild of the qemu.spec.
> 
> Is this not possible with debian packages?  Aren't there at least
> separate ${lib} and ${lib}-devel packages?
-----------------------------------------------------------------------

Then I:
> I think we are talking at cross purposes. In Debian a single source package
> does indeed build $lib and $lib-dev etc.
> 
> But, those packages are for arch={i386,amd64,armhf}. They are not for
> arch={stubdom-i386,stubdom-amd64} and there is no such arch in Debian (nor
> CentOS nor Fedora I expect).
> 
> For the qemu source package to be able build a runnable qemu-xenpv binary
> it would need to get all the libFOO and libBAR which qemu needs from
> somewhere, and they would need to be built for the stubdom-{i386,amd64}
> arches, not for i386 or amd64. All the builddeps for qemu-pv need to come
> from somewhere and the existing i386 or amd64 libraries do not satisfy the
> build dep for a pv stubdom.
> 
> (To be clear, I'm talking about mini-os/rump kernel stub doms here, not
> Linux ones of any sort)
-----------------------------------------------------------------------

To which George said:
> I think I see what you're saying -- if building a qemu stubdom *only*
> involved the qemu codebase itself, then it would be really easy for the
> qemu package maintainers to add qemu-xenpv (or something) which would
> just build qemu to run in PV mode.  But in fact, building qemu stubdoms
> involves building all of the other libraries upon which qemu depends for
> stubdoms as well; which again, we don't want duplicated inside the qumu
> package, and so would mean re-building all those other packages for
> stubdoms as well.
> 
> Interestingly, if we cast a vision for a world where unikernels are the
> norm, then it might actually make sense for a distro like Debian to go
> through and re-build core libraries against rump kernels, so that then
> applications like qemu (or apache, or whatever) could link against them
> and build a bootable image, perhaps capable of running either on Xen or
> KVM.
> 
> That's obviously not going to happen overnight -- but it would be
> interesting for Someone to give a try, just to see how difficult it is.
>  If it's not that difficult to build libraries against rumpkernels and
> package them up so they can be linked to from applications, then the
> road to Debian / Fedora-built unikernel applications (including qemu)
> may not actually be that long.
-----------------------------------------------------------------------

Wei and I both replied. Wei said:
> The vision is that rump kernel toolchains will be packaged to distros,
> so that people can build rump kernel with all necessary source tarballs.
> That is not exactly the same as what you just said. What you said is
> probably next step down the road.
> 
> > That's obviously not going to happen overnight -- but it would be
> > interesting for Someone to give a try, just to see how difficult it
is.
> 
> Most faffs will be fighting with build systems of individual packages.
> Also some software has less tested configuration that doesn't actually
> work on source code level. For example, I'm very well aware some QEMU
> bugs that prevents it from building with rump kernel.
> 
> All those fixes to build systems and source code can only happen as
> developers realise rump kernel is a thing that they care about. It's a
> bit unrealistic to expect a single team to fix every single piece of
> software.
and I said:
> The first hurdle would be how much of the distro is cross-compilable, since
> rump kernels necessarily imply that.
> 
> That in turn requires the rumpkernel toolchain to look like any other cross
> toolchain i.e. you call -gcc, otherwise every package would need
> tweaking to use the rumpkernel build system.
> 
> Probably that then implies a load of config.{sub,guess} updates for every
> package etc but those are normal for crossing any new architecture and I
> think are reasonably well understood these days.
-----------------------------------------------------------------------

At this point we decided this should go on list, and here we are...

>>> On 08.09.15 at 11:24, <ian.campbell at citrix.com> wrote:
> Release cycle
> ============> 
> Waldi commented that the stable release cycle was too long. Would like
> to see a release after any large security update.
> 
> We asked if the RCs for stable releases were valuable, the answer was
> "not so much".
> 
> Waldi would prefer to avoid cherry-picking security fixes if possible.
> 
> We asked if we thought Xen stable releases could be added to Debian
> point releases. Waldi thought they likely could be, citing the
> inclusion of Linux stable releases in point releases.
> 
> Our stable releases follow a similar set of rules to Linux, we think
> we implement them more faithfully (less feature or feature-like
> backports)
> 
> ACTION: Talk to Jan about making changes to stable release process.
That's kind of the opposite of what we quite recently changed to
(a [hopefully] more predictable four month cycle). Apart from the
question what "large" is, doing a release after any large security
update seems unreasonable to me (not only because of giving up
the predictability, but also because of the overhead involved,
which is there even if we ditched the RCs). I have to admit that I
fail to see why Debian would be different than other distros, all
cherry picking security fixes until a new stable release becomes
available. If otoh other major distros voiced similar desires, I
think we'd have to once again re-think our stable release cadence.

Jan

> On 8 Sep 2015, at 10:47, Jan Beulich <jbeulich at suse.com> wrote:
> 
>>>> On 08.09.15 at 11:24, <ian.campbell at citrix.com> wrote:
>> Release cycle
>> ============>> 
>> Waldi commented that the stable release cycle was too long. Would like
>> to see a release after any large security update.
>> 
>> We asked if the RCs for stable releases were valuable, the answer was
>> "not so much".
>> 
>> Waldi would prefer to avoid cherry-picking security fixes if possible.
>> 
>> We asked if we thought Xen stable releases could be added to Debian
>> point releases. Waldi thought they likely could be, citing the
>> inclusion of Linux stable releases in point releases.
>> 
>> Our stable releases follow a similar set of rules to Linux, we think
>> we implement them more faithfully (less feature or feature-like
>> backports)
>> 
>> ACTION: Talk to Jan about making changes to stable release process.
> 
> That's kind of the opposite of what we quite recently changed to
> (a [hopefully] more predictable four month cycle). Apart from the
> question what "large" is, doing a release after any large
security
> update seems unreasonable to me (not only because of giving up
> the predictability, but also because of the overhead involved,
> which is there even if we ditched the RCs).
I suppose the question is what the real problem for distros is: 

If it is the process of cherry picking and merging, then we could create a tag
on a stable branch more frequently (e.g. every month), which makes it easier for
distros to consume a number of security fixes, while not creating all the
overhead of creating a release.

Whether such a tagged stable branch is an RC (without a tarball) or not, would
be a different question.
> I have to admit that I
> fail to see why Debian would be different than other distros, all
> cherry picking security fixes until a new stable release becomes
> available. If otoh other major distros voiced similar desires, I
> think we'd have to once again re-think our stable release cadence.
A fully tested maintenance release, created more frequently would be a lot more
challenging and require a lot more effort.
I do agree with Jan that we ought to approach other distros and then re-think. 

Lars

On Tue, 2015-09-08 at 03:47 -0600, Jan Beulich wrote:
> > > > On 08.09.15 at 11:24, <ian.campbell at citrix.com>
wrote:
> > Release cycle
> > ============> > 
> > Waldi commented that the stable release cycle was too long. Would like
> > to see a release after any large security update.
> > 
> > We asked if the RCs for stable releases were valuable, the answer was
> > "not so much".
> > 
> > Waldi would prefer to avoid cherry-picking security fixes if possible.
> > 
> > We asked if we thought Xen stable releases could be added to Debian
> > point releases. Waldi thought they likely could be, citing the
> > inclusion of Linux stable releases in point releases.
> > 
> > Our stable releases follow a similar set of rules to Linux, we think
> > we implement them more faithfully (less feature or feature-like
> > backports)
> > 
> > ACTION: Talk to Jan about making changes to stable release process.
> 
> That's kind of the opposite of what we quite recently changed to
> (a [hopefully] more predictable four month cycle). Apart from the
> question what "large" is, doing a release after any large
security
> update seems unreasonable to me (not only because of giving up
> the predictability, but also because of the overhead involved,
> which is there even if we ditched the RCs). I have to admit that I
> fail to see why Debian would be different than other distros, all
> cherry picking security fixes until a new stable release becomes
> available. If otoh other major distros voiced similar desires, I
> think we'd have to once again re-think our stable release cadence.
IIRC (hopefully Ian or someone else will correct me if not) the main
proposal to discuss with you was WRT the usefulness of the RCs for stable
releases, since it was felt they didn't provide much benefit to
downstreams.

IOW perhaps it would be just as useful to downstreams and less work for us
(mainly you I suppose) to do 0 or only 1 rc for a point release, based on
whatever is in the branch at the appropriate time. It might also avoid
various delays which the stable release process can currently suffer from
waiting for a push for each rc instead of just once (or maybe twice), which
in turn might help the release timing to be even more predictable.

Ian.

Hi,

Wei Liu hinted that I should "chime in and / or provide corrections" 
(his words).  I'll attempt to do exactly that by not really replying to 
anything specific.  For the record, when I say "we" in this mail, I
mean
"people who have contributed to the rump kernel project" (as also 
indicated by the email-hat).

First of all, there's a difference between a rump kernel (driver bundle 
built out of unmodified kernel components) and any unikernel you 
construct out of rump kernels ... sort of like how there's a difference 
between Linux and GNU/Linux.

For unikernels, the rump kernel project provides Rumprun, which can 
provide you with a near-full POSIX'y interface.  Rumprun also provides 
toolchain wrappers so that you can compile existing programs as Rumprun 
unikernels.  Rumprun also recently regrew the ability to run without the 
POSIX'y bits; some people found it important to be able to make a 
tradeoff between running POSIX'y applications and more compact "kernel 
plane" unikernels such as routers and firewalls.  But, for brevity and 
simplicity, I'll assume the POSIX'y mode for the rest of this email, 
since that's what the QEMU stubdom will no doubt use.

If the above didn't explain the grand scheme of things clearly, have a 
look at http://wiki.rumpkernel.org/Repo and especially the picture.  If 
things are still not clear after that, please point out matters of 
confusion and I will try to improve the explanations.

Also for simplicity, I'll be talking about rump kernels constructed from 
the NetBSD kernel, and the userspace environment of Rumprun being 
NetBSD-derived.  Conceptually, there's nothing stopping someone from 
plugging a GNU layer on top of NetBSD-derived rump kernels (a bit like 
Debian kXBSD?) or constructing rump kernels out of Linux.  But for now, 
let's talk about the only working implementation.

As far as I know, the API/ABI of the application environment provided by 
Rumprun is the same as the one provided by standard NetBSD.  Granted, I 
didn't perform the necessary experiments to verify that, so take the 
following with a few pinches of salt.  In theory, you could take 
application objects built for NetBSD and link them against Rumprun libs. 
  However, since a) nobody (else) ships applications as relocatable 
static objects b) Rumprun does not support shared libraries, I don't 
know how helpful the fact of ABI compatibility is.  IMO, adding shared 
library support would be a backwards way to go: increasing runtime 
processing and memory requirements to solve a build problem sounds plain 
weird.  So, I don't think you can leverage anything existing.

We do have most of the Rumprun cross-toolchain figured out at this 
point.  First, we don't ship any backend toolchain(s), but rather bolt 
wrappers and specs on top of any toolchain (*) you provide.  That way we 
don't have to figure out where to get a toolchain which produces binary 
for every target that everyone might want.  Also, it makes bootstrapping 
Rumprun convenient, since you just say "hey give me the components and 
application wrappers for CC=foocc" and off you go.

*) as long as it's gcc-derived, for now (IIRC gcc 4.8 - 5.1 are known to 
work well, older than that at least C++ won't work).  clang doesn't 
support specs files at least AFAIK, so someone would have to figure out 
how to move the contents of the specs into the wrappers, or whatever 
equivalent clang uses.  (patches welcome ;)

The produced wrappers look exactly like a normal cross-toolchain.  The 
tuple is the same as what NetBSD uses, except with rumprun added in the 
middle, so e.g. x86_64-rumprun-netbsd or arm-rumprun-netbsdelf-eabihf. 
That naming scheme means that most GNU-using software compiles nicely 
for Rumprun just by running configure as ./configure 
--host=x86_64-rumprun-netbsd followed by "make".  Sometimes you 
additionally need things like --disable-shared, but all in all 
everything works pretty well.  See 
http://repo.rumpkernel.org/rumprun-packages for a bunch of "case 
studies", not limited to just GNU autotools.

After "make", before launch we have an additional step called
"bake",
which links the specific kernel components onto the binary.  So for 
example, you can make the compiled binary run on Xen or KVM depending on 
which kernel components you bake onto it.  As a crude analogy, it's like 
scp'ing a binary to a Xen or KVM or bare metal system, but since the 
Rumprun unikernel is missing exec, we use the linker to perform "system 
selection".

So for shipping, one option is to ship the binary after "make", but
then
you also need to ship the toolchain.  The other option is to ship the 
baked binary, but then you lose some of your possibilities on how to 
target the binary.  I'm not sure either option is right for all cases.

We're still trying to figure out the exact form and figure of 
bake+launch.  In the original implementation we assumed that at 
launch-time we could cheaply control all of the details of the backend 
(a la xl.conf).  That assumption proved to be bad not only for example 
for embedded systems (which we should've foreseen), but also for cases 
like Amazon EC2 (where creating something launchable and launching 
something are seriously separate steps).  I'm not going to go into 
details in this thread, but just saying that we still have a few things 
to figure out in the full source-to-execution chain.  If someone wants 
to ship something, please please check with the rump kernel community 
first so that we know that you want to depend on some syntax/semantics 
not changing anymore.

I don't really have good solutions for the packaging problem.  Building 
a "full distro" around rump kernels certainly sounds interesting, and 
we're sort of trying to experiment with that with rumprun-packages. 
However, for packages sooner or later we need to assimilate a real 
packaging system which properly manages dependencies, licenses, 
vulnerabilities, etc.  I'm unsure we'd want to assimilate every existing
packaging system -- Rumprun works the same on every platform, after all. 
  Hard to say for now, I hadn't actually considered the case where a 
distro might want to natively ship binary Rumprun images, as opposed to 
just the toolchain and components.

If I were you folks, I'd start by getting qemu out of the door, and 
worry about generalized solutions when someone wants to ship the second 
unikernel (or third or any small N>1).  If you can't sell to distros 
something that solves a problem, it's unlikely you'll be able to sell a 
framework for solving the same one problem (though, granted, it might be 
easier to sell a framework to computer folk -- "nevermind the solution, 
here's abstraction!").

If you haven't tried rumprun-packages, I recommend to pick something 
from there and see how it works.  Doing so might give some more 
perspective on how easy or difficult it would be to package QEMU.

Whoops, ended up being a bit longer than what I hoped for, but with any 
luck at least some new information was communicated.

   - antti

On Tue, 2015-09-08 at 10:24 +0100, Ian Campbell wrote:
> Embedding in xen.git
> ===================> 
> We are much better about providing ways to use system-supplied
> components these days (since 4.4) and Debian uses them.
> 
> Waldi noted that iPXE did not have such an option. Since that iPXE is
> only used by qemu-trad (for qemu-upstream the iPXE comes from the PCI
> ROM BAR) and Debian disabled qemu-trad it should be pretty quick to
> patch the build system to disable iPXE.
I went to do this, and found that ipxe is already only built if ROMBIOS is
enabled and ROMBIOS is disabled by default if qemu-trad is not enabled
(which it is not in the Debian packaging).

So I think this is already addressed! I don't think it is worth at this
point allowing to control ipxe separately from ROMBIOS (or at least such a
task is nowhere near the top of my todo list).

Ian.

On Tue, 2015-09-08 at 10:24 +0100, Ian Campbell wrote:
> grub-xen
> =======> 
> Needs much better docs.
> 
> ACTION: I agreed to move the text of my blog post somewhere more
> obvious.
I did this on the last doc day: http://wiki.xen.org/wiki/PvGrub2

Ian.