Pkg xen devel - Nov 2020 - Bug#975062: Python 3 (pygrub) in 4.14 packages

I was trying to test `pygrub` and found the Python 3 version is
definitely broken in the 4.14 packages.  I was able to get the script to
display the help message by adding
"/usr/lib/xen-4.14/lib/<triplet>" to
sys.path.

The existing line:
sys.path.insert(1, sys.path[0] + '/../lib/python')

Is distinctly odd, usually this is better expressed:
sys.path.append(os.path.join(sys.path[0], "libexec"))

(though I suppose we can assume Linux, but this is Bad Practice)

The way some portions of `pygrub` are packaged are distinctly odd.
Certainly xc.so and xs.so are linked to core Xen libraries and need to be
version-specific.  Yet libfsimage.so appears independant of the Xen
version and should likely track `pygrub`, rather than matching the system
Xen version.

I may get a chance to chase things down better later.


In other news, Julien Grall's latest set of 4 patches are valuable on ARM
ACPI systems (RP4 with Tianocore) and having those in the next Debian
will be be high-value.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg at m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Package: src:xen
Version: 4.14.0-1~exp1
Control: submitter -1 ehem+debian at m5p.com
X-Debbugs-CC: ehem+debian at m5p.com, ijackson at chiark.greenend.org.uk

Hi, I think this should be in a bug report in the BTS to track it in a
better way.

----8<---- Forwarded Message ----8<----
Subject: [Pkg-xen-devel] Python 3 in 4.14 packages
Date: Sat, 26 Sep 2020 22:44:25 -0700
From: Elliott Mitchell <ehem+debian at m5p.com>
To: pkg-xen-devel at alioth-lists.debian.net

I was trying to test `pygrub` and found the Python 3 version is
definitely broken in the 4.14 packages.  I was able to get the script to
display the help message by adding
"/usr/lib/xen-4.14/lib/<triplet>" to
sys.path.

The existing line:
sys.path.insert(1, sys.path[0] + '/../lib/python')

Is distinctly odd, usually this is better expressed:
sys.path.append(os.path.join(sys.path[0], "libexec"))

(though I suppose we can assume Linux, but this is Bad Practice)

The way some portions of `pygrub` are packaged are distinctly odd.
Certainly xc.so and xs.so are linked to core Xen libraries and need to be
version-specific.  Yet libfsimage.so appears independant of the Xen
version and should likely track `pygrub`, rather than matching the system
Xen version.

---->8---- Forwarded Message ---->8----

I also have a little snippet from IRC, which is about this, where Ian
reports that he's seen it working.

https://salsa.debian.org/xen-team/debian-xen/-/snippets/500

So, apparently there are cases in which pygrub 'works' and in which it
does not, and apparently using pygrub with "amd64 kernel and Xen tools
but i386 userland" is problematic, and I remember some remarks which I
can't find back about that that use case was probably already broken
always, in the past.

I wanted to find out about this and set up some test cases to reproduce
things (I've never used pygrub yet), but that obviously did not happen
yet. I have some stuff going on in my personal life that is taking up a
lot of time currently. What is rather easy for *me* is to help
organizing the work and managing todo lists etc, but not learning new
stuff ATM.

So, my current questions are:

1. Is pygrub a blocker for having Xen 4.14 in unstable? Because that
should be our first team-goal now.
2. What exactly is going on, can we make a list/table/whatever about in
which cases pygrub 'does not work' (in more detail, how does it fail).
3. pygrub keeps being the thing that always causes problems. What would
be your (asking anyone who wants to think along) ideas about which
well-defined situations/test-cases we should have to execute instead of
having the users report problems after big package changes?

Hans

P.S. Next message after the commercials will be on #968965 which is the
other biggest issue for Xen 4.14 in unstable now.

Processing control commands:
> submitter -1 ehem+debian at m5p.com
Bug #975062 [src:xen] Python 3 (pygrub) in 4.14 packages
Changed Bug submitter to 'ehem+debian at m5p.com' from 'Hans van
Kranenburg <hans at knorrie.org>'.

-- 
975062: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=975062
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems

On Wed, Nov 18, 2020 at 04:32:00PM +0100, Hans van Kranenburg
wrote:
> I also have a little snippet from IRC, which is about this, where Ian
> reports that he's seen it working.
> 
> https://salsa.debian.org/xen-team/debian-xen/-/snippets/500
> 
> So, apparently there are cases in which pygrub 'works' and in which
it
> does not, and apparently using pygrub with "amd64 kernel and Xen tools
> but i386 userland" is problematic, and I remember some remarks which I
> can't find back about that that use case was probably already broken
> always, in the past.
> 
> I wanted to find out about this and set up some test cases to reproduce
> things (I've never used pygrub yet), but that obviously did not happen
> yet. I have some stuff going on in my personal life that is taking up a
> lot of time currently. What is rather easy for *me* is to help
> organizing the work and managing todo lists etc, but not learning new
> stuff ATM.
> 
> So, my current questions are:
> 
> 1. Is pygrub a blocker for having Xen 4.14 in unstable? Because that
> should be our first team-goal now.
> 2. What exactly is going on, can we make a list/table/whatever about in
> which cases pygrub 'does not work' (in more detail, how does it
fail).
> 3. pygrub keeps being the thing that always causes problems. What would
> be your (asking anyone who wants to think along) ideas about which
> well-defined situations/test-cases we should have to execute instead of
> having the users report problems after big package changes?
> 
> Hans
> 
> P.S. Next message after the commercials will be on #968965 which is the
> other biggest issue for Xen 4.14 in unstable now.
Due to working with Pry Mar, I can state the cross-compilation of the
Python shared objects may not be 100% functional yet.  Looks very much
like Python's "distutils" took 2 steps forward and then 2 steps
backward
during the Python 2 -> Python 3 transition.

(Great!  Linking and compilation got separated.  Ewww!  CFLAGS gets
appended to LDFLAGS.  Great!  We'll add support for architecture-specific
compilation directories.  Ewww!  There isn't a good way to pass in the
architecture triplet.)

I've got an initial patch for working around an issue here, but the
quality doesn't look great to me.  Something along those lines should be
submitted to Xen, but I'm unsure of all the issues.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg at m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445


-------------- next part --------------
A non-text attachment was scrubbed...
Name: tools-python-Correct-extension-filenames-for-Python-.patch
Type: text/x-diff
Size: 2780 bytes
Desc: not available
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20201118/1ff902a8/attachment-0002.patch>

Hans van Kranenburg writes ("Bug#975062: Python 3 (pygrub) in 4.14
packages"):
> So, apparently there are cases in which pygrub 'works' and in which
it
> does not, and apparently using pygrub with "amd64 kernel and Xen tools
> but i386 userland" is problematic, and I remember some remarks which I
> can't find back about that that use case was probably already broken
> always, in the past.
The problem with pygrub with 32-bit userland is as follows:

 * Xen has to be 64-bit since there is no 64-bit Xen.

 * dom0 kernel bitness and Xen tools bitness must match because
   Xen 32/64 compat ABI understands only one bitness for dom0
   and Xen dom0 tools make hypercalls directly so must match
   the kernel.

 * 32-bit kernels are starting not to be able to drive hardware
   (big PCI bars, bugs, etc.) so you want a 32-bit kernel.

 * So you must have libxen*:amd64.

 * pygrub uses python, obviously.  It needs to load the xenfsimage
   library, which is part of xen tools, since that is the userland
   library tht understands the guest filesystem to fish out the guest
   kernel.

 * The xenfsimage library is not in its own package [1] - it's in with
   some other Xen libraries.  But it is going to be loaded into a
   python interpreter, so it needs to match the bitness of the python
   interpreter.

 * You can't install a 64-bit python interpreter without basically
   doing the whole 32-to-64 crossgrade.  (That crossgrade is what I
   ended up doing on my home machine.)

 * You can't co-install libxen*:i386 because the Xen libraries aren't
   properly multiarched. [2]

 * This gets worse now that the Xen packages use python3.  Previously
   with a minimal but modern system you might be able to get away with
   having a 64-bit python2 and a 32-bit python3.

Both [1] and [2] are in principle bugs in the Xen packages.  Upstream
sentiment seems to be that 32-bit userland is not really a very good
idea any more anyway.  So we could solve this by fixing [1] or [2] or
we could expect people to use 64-bit dom0 userland (and crossgrade if
need be).
> I wanted to find out about this and set up some test cases to reproduce
> things (I've never used pygrub yet), but that obviously did not happen
> yet. I have some stuff going on in my personal life that is taking up a
> lot of time currently. What is rather easy for *me* is to help
> organizing the work and managing todo lists etc, but not learning new
> stuff ATM.
> 
> So, my current questions are:
> 
> 1. Is pygrub a blocker for having Xen 4.14 in unstable? Because that
> should be our first team-goal now.
I think yes, a working pygrub ought to be a blocker for 4.14 in
unstable.

But I think we have that - I rebuilt the existing packages for buster
and it WFM.
> 2. What exactly is going on, can we make a list/table/whatever about in
> which cases pygrub 'does not work' (in more detail, how does it
fail).
> 3. pygrub keeps being the thing that always causes problems. What would
> be your (asking anyone who wants to think along) ideas about which
> well-defined situations/test-cases we should have to execute instead of
> having the users report problems after big package changes?
IDK about any other problems than the bitness one above.

Ian.

-- 
Ian Jackson <ijackson at chiark.greenend.org.uk>   These opinions are my
own.

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

It seems I was distracted when I wrote this mail.

Ian Jackson writes ("Re: Bug#975062: Python 3 (pygrub) in 4.14
packages"):
> The problem with pygrub with 32-bit userland is as follows:
> 
>  * Xen has to be 64-bit since there is no 64-bit Xen.
                                            ^^ 32
>  * 32-bit kernels are starting not to be able to drive hardware
>    (big PCI bars, bugs, etc.) so you want a 32-bit kernel.
                                              ^^ 64


-- 
Ian Jackson <ijackson at chiark.greenend.org.uk>   These opinions are my
own.

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

Hi!

On 11/18/20 6:45 PM, Ian Jackson wrote:
> Hans van Kranenburg writes ("Bug#975062: Python 3 (pygrub) in 4.14
packages"):
>> So, apparently there are cases in which pygrub 'works' and in
which it
>> does not, and apparently using pygrub with "amd64 kernel and Xen
tools
>> but i386 userland" is problematic, and I remember some remarks
which I
>> can't find back about that that use case was probably already
broken
>> always, in the past.
> 
> The problem with pygrub with 32-bit userland is as follows:
> 
>  * Xen has to be 64-bit since there is no 64-bit Xen.
                                            ^^ 32?
>  * dom0 kernel bitness and Xen tools bitness must match because
>    Xen 32/64 compat ABI understands only one bitness for dom0
>    and Xen dom0 tools make hypercalls directly so must match
>    the kernel.
> 
>  * 32-bit kernels are starting not to be able to drive hardware
>    (big PCI bars, bugs, etc.) so you want a 32-bit kernel.
                                              ^^ 64?
>  * So you must have libxen*:amd64.
> 
>  * pygrub uses python, obviously.  It needs to load the xenfsimage
>    library, which is part of xen tools, since that is the userland
>    library tht understands the guest filesystem to fish out the guest
>    kernel.
> 
>  * The xenfsimage library is not in its own package [1] - it's in with
>    some other Xen libraries.  But it is going to be loaded into a
>    python interpreter, so it needs to match the bitness of the python
>    interpreter.
> 
>  * You can't install a 64-bit python interpreter without basically
>    doing the whole 32-to-64 crossgrade.  (That crossgrade is what I
>    ended up doing on my home machine.)
> 
>  * You can't co-install libxen*:i386 because the Xen libraries
aren't
>    properly multiarched. [2]
> 
>  * This gets worse now that the Xen packages use python3.  Previously
>    with a minimal but modern system you might be able to get away with
>    having a 64-bit python2 and a 32-bit python3.
> 
> Both [1] and [2] are in principle bugs in the Xen packages.  Upstream
> sentiment seems to be that 32-bit userland is not really a very good
> idea any more anyway.  So we could solve this by fixing [1] or [2]
My personal opinion is that there are more interesting horses to fry (or
what was the saying) for our very bandwidth limited team. So yes, let's
move this from the perfection into the known issues department for now
(bullseye).
> or
> we could expect people to use 64-bit dom0 userland (and crossgrade if
> need be).
When someone shows up with a real world issue who's really panicing
after recklessly upgrading (after the bullseye release halfway 2021) we
probably might help by giving pointers and instructions. Until that
actually happens, we should not spend time writing documentation about
how to do that etc.
>> I wanted to find out about this and set up some test cases to reproduce
>> things (I've never used pygrub yet), but that obviously did not
happen
>> yet. I have some stuff going on in my personal life that is taking up a
>> lot of time currently. What is rather easy for *me* is to help
>> organizing the work and managing todo lists etc, but not learning new
>> stuff ATM.
>>
>> So, my current questions are:
>>
>> 1. Is pygrub a blocker for having Xen 4.14 in unstable? Because that
>> should be our first team-goal now.
> 
> I think yes, a working pygrub ought to be a blocker for 4.14 in
> unstable.
> 
> But I think we have that - I rebuilt the existing packages for buster
> and it WFM.
OK.
>> 2. What exactly is going on, can we make a list/table/whatever about in
>> which cases pygrub 'does not work' (in more detail, how does it
fail).
>> 3. pygrub keeps being the thing that always causes problems. What would
>> be your (asking anyone who wants to think along) ideas about which
>> well-defined situations/test-cases we should have to execute instead of
>> having the users report problems after big package changes?
> 
> IDK about any other problems than the bitness one above.
Ok, thanks a lot for the write up, both, and now we have a debian bug to
look back to which is a bit easier to track than mailing list messages.

So, I'm crossing out this issue now as blocker.

Hans

On 11/18/20 9:39 PM, Ian Jackson wrote:
> It seems I was distracted when I wrote this mail.
> 
> Ian Jackson writes ("Re: Bug#975062: Python 3 (pygrub) in 4.14
packages"):
>> The problem with pygrub with 32-bit userland is as follows:
>>
>>  * Xen has to be 64-bit since there is no 64-bit Xen.
>                                             ^^ 32
> 
>>  * 32-bit kernels are starting not to be able to drive hardware
>>    (big PCI bars, bugs, etc.) so you want a 32-bit kernel.
>                                               ^^ 64
> 
HAH, thanks. \:D/

Hans