Pkg xen devel - Sep 2020 - [PATCH 0/3] More work towards viable Xen cross-builds

I guess we're starting a series of you (Hans van Kranenburg) do some
updates, and then I do some updates.

The first patch is a small packaging bug.  The Xen build needs Python
for 2 distinct purposes.  First, it is needed for portions of the build
process itself (notably building the runtime Python
interface/"distutils").  Second, it is needed for linking the runtime
portion together.  The first needs *any* variant of the python3-dev
package (build machine variant is fine), second needs the host machine
libpython3-dev package.  Having seen other packages, this is
"python3-dev:any" and "libpython3-dev", *not*
"python3-dev".

Next is a portion I've passed to xen-devel.  When doing cross-building
the compiler/linker selection is crucial.  The default search for them
will get the build versions, rather than the host versions.  Similar to
the Debian package, the upstream source wasn't passing the chosen linker
to the Python build and hence was attempt to link for the build machine
rather than the host machine.  Also turned out linker options for
linking an executable were being passed instead of for linking a shared
object.

Last patch is an update of the "Disable OCAML build when
cross-building"
patch.  During some searching I ran across the "Debian Build Profile
Specification" (https://wiki.debian.org/BuildProfileSpec) this covers one
of the OCAML issues I've been running into.  Notably the dependancy on
ocaml-nox/ocaml-findlib should only be enforced if *not* cross-building.

The actual mechanism is the profile spec has a "noocaml" profile.  If
this is active, simply skip the OCAML portions or for the two expected
files create empty ones.  Question is how bug #774129 will be fixed?
Potentially there is a need to turn this on in presence of the "cross"
profile, but `dpkg-dev` could readily take care of this for Xen and most
other packages, but #774129 may not be resolved this way.


Given the situation it really looks like the Xen packages should be
broken up further.  Several pieces included in the main packages are
optional or have other options.  oxenstored is one such extra which is
pretty well entirely optional.

Notably pvgrub is rather better than pygrub in most situations, yet
pygrub remains built-in and you have to search for pvgrub.  I would tend
to propose creating a virtual package "xen-domu-bootloader" and have
"xen-pygrub" provide that (instead of xen-utils-X.Y directly
recommending, what if something else showed up?).


Some of the recommendations on the Debian Wiki appear not to match
reality.  Reducing the number of vCPUs given to Domain 0 on start reduces
some overhead on Domain 0.  On the flip side Xen won't initialize ACPI
C-states on processors which lack an associated Domain 0 vCPU.  Also
need to modify the init script to load xen_acpi_processor.


Elliott Mitchell (3):
  debian/control: Fix python dependancy
  tools/python: Pass linker to Python build process
  debian/rules: Setup use of noOCAML profile

 debian/control        |  7 ++++---
 debian/rules          | 17 +++++++++++++++--
 tools/pygrub/Makefile |  8 ++++----
 tools/python/Makefile |  9 +++++----
 4 files changed, 28 insertions(+), 13 deletions(-)


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg at m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Hi!

On 9/14/20 3:43 AM, Elliott Mitchell wrote:
> I guess we're starting a series of you (Hans van Kranenburg) do some
> updates, and then I do some updates.
> 
> The first patch is a small packaging bug.  The Xen build needs Python
> for 2 distinct purposes.  First, it is needed for portions of the build
> process itself (notably building the runtime Python
> interface/"distutils").  Second, it is needed for linking the
runtime
> portion together.  The first needs *any* variant of the python3-dev
> package (build machine variant is fine), second needs the host machine
> libpython3-dev package.  Having seen other packages, this is
> "python3-dev:any" and "libpython3-dev", *not*
"python3-dev".
Aha. Thanks for the explanation. I didn't completely get this before and
tried something that at least made it BFS.

Applied with s/dependan/dependen/, thanks.
> Next is a portion I've passed to xen-devel.  When doing cross-building
> the compiler/linker selection is crucial.  The default search for them
> will get the build versions, rather than the host versions.  Similar to
> the Debian package, the upstream source wasn't passing the chosen
linker
> to the Python build and hence was attempt to link for the build machine
> rather than the host machine.  Also turned out linker options for
> linking an executable were being passed instead of for linking a shared
> object.
Ok, this is "[PATCH] tools/python: Pass linker to Python build
process"
and I see there is no reaction yet. Ian is one of the maintainers of
tools/ upstream, so maybe he can look at it in the context of these
debian packaging changes?

Basic rule is that we only want to add upstream patches to the packaging
if they're accepted so that we're sure that we can get rid of the exact
same thing in the future again.
> Last patch is an update of the "Disable OCAML build when
cross-building"
> patch.  During some searching I ran across the "Debian Build Profile
> Specification" (https://wiki.debian.org/BuildProfileSpec) this covers
one
> of the OCAML issues I've been running into.  Notably the dependancy on
> ocaml-nox/ocaml-findlib should only be enforced if *not* cross-building.
> 
> The actual mechanism is the profile spec has a "noocaml" profile.
If
> this is active, simply skip the OCAML portions or for the two expected
> files create empty ones.  Question is how bug #774129 will be fixed?
> Potentially there is a need to turn this on in presence of the
"cross"
> profile, but `dpkg-dev` could readily take care of this for Xen and most
> other packages, but #774129 may not be resolved this way.
For some weird reason 3/3 never ended up in my mailbox. I found it at
https://alioth-lists.debian.net/pipermail/pkg-xen-devel/2020-September/008182.html

Ok, I see that this one replaces "debian/rules: Disable OCAML build when
cross-building", that's fine.

I just tried to remove the previous patch and replace it at the same
point after some gunzip and copy paste and git am magic, but after
spending more than an hour trying to apply a single patch and ending up
with weird errors like "fatal: empty ident name (for <>) not
allowed"...
please get a salsa gitlab account if you don't already have it and push
your branches right into the packaging repo under your nickname.

Sending patch mail blasts is for the purpose of review and discussion.
Actually applying them is easier when I already have them in git in one
of your branches.

I don't have all the puzzle pieces together yet in my mind to understand
the change, so I'll just do some thinking out loud so you can correct
me. noocaml has the description 'Disable ocaml bindings', but oxenstored
is not some bindings, it's an actual executable. But all of that does
not matter, since you explicitly decide where it's going to be applied
by making use of filter noocaml. What I'm missing now is how this
noocaml gets activated, and by what.

Reading 774129, I understand the 'once'..'will then need to
reenable'
and why that is undesirable. But if it helps the case now, we can do
whatever makes it work now in here. When dpkg-* gets fixed, either
you're around to make further changes, or someone else who cares and can
test the changes will do it. To support the second case, please also
move some relevant lines from the cover letter (like, refering to
774129) to the actual commit message, since the 0/ cover letter will
never make it to the git history, but the Debian bug history will
probably be around.
> Given the situation it really looks like the Xen packages should be
> broken up further.  Several pieces included in the main packages are
> optional or have other options.  oxenstored is one such extra which is
> pretty well entirely optional.
For Buster, we switched to oxenstored as default (while cxenstored still
gets built also), and there has not been any report about it failing to
do its work yet. It's not an optional extra thing, it's the default
recommended thing now.

Currently in the init script that starts xenstored, we look around for a
list of xenstored binaries, and start the first one found. This is
because the debian xen init script is in the commons package, and it
needs to be able to run on a system that is rebooted back into a
previous Xen version after doing an OS upgrade to Buster. This is
relevant for the Stretch->Buster upgrade.

Since I think we should limit ourselves to only support N to N+1
upgrades as a rule, I'm totally open to further changes and only make
sure that upgrading from Buster to whatever we do now will work properly
(and of course rolling upgrades in testing/unstable meanwhile).

Still building the c xenstored by default and shipping it anywhere is
not a problem at all I think, if we have cases where this makes it
easier to just omit oxenstored and have it use c xenstored then. (?)
> Notably pvgrub is rather better than pygrub in most situations, yet
> pygrub remains built-in and you have to search for pvgrub.  I would tend
> to propose creating a virtual package "xen-domu-bootloader" and
have
> "xen-pygrub" provide that (instead of xen-utils-X.Y directly
> recommending, what if something else showed up?).
Yeah. I personally never have used pygrub, but always have been using
pvgrub to start PV domUs and have been working with Juergen to make the
PVH case work. And now, at work, all the things are booting with pvgrub2
in PVH mode, with a hardcoded config in the grub2 image since kernel and
initrd are always symlinked anyway in the Debian domU filesystem, so no
grub whatever in the domU is needed. The Debian grub-xen package stuff
needs a grub.cfg in the domU.

https://salsa.debian.org/knorrie/pvgrub2

Also, this has been in my configs forever:

APT::Install-Recommends "false";
APT::Install-Suggests "false";

and I have never regretted doing that.
> Some of the recommendations on the Debian Wiki appear not to match
> reality.
Yes, I would rather see distro specific wiki pages mostly disappear and
instead have contributions to upstream wiki pages unless it's *really*
something *really* distro specific. For a distro wiki it's just too easy
to cobble up some recommendations because some problem existed years ago
and then have users think this is really important to do years later
when the problem is already fixed long time...
> Reducing the number of vCPUs given to Domain 0 on start reduces
> some overhead on Domain 0.  On the flip side Xen won't initialize ACPI
> C-states on processors which lack an associated Domain 0 vCPU.  Also
> need to modify the init script to load xen_acpi_processor.
That's interesting.
> Elliott Mitchell (3):
>   debian/control: Fix python dependancy
>   tools/python: Pass linker to Python build process
>   debian/rules: Setup use of noOCAML profile
> 
>  debian/control        |  7 ++++---
>  debian/rules          | 17 +++++++++++++++--
>  tools/pygrub/Makefile |  8 ++++----
>  tools/python/Makefile |  9 +++++----
>  4 files changed, 28 insertions(+), 13 deletions(-)
thanks,
Hans