Pkg xen devel - Sep 2021 - Bug#991967: Simply ACPI powerdown/reset issue?

On Tue, Sep 21, 2021 at 06:33:20AM -0400, Chuck Zmudzinski
wrote:
> I presume you are suggesting I try booting 4.19.181-1 on the
> current version of Xen-4.14 for bullseye as a dom0. I am not
> inclined to try it until an official Debian developer endorses
> your opinion that the bug I am seeing is distinct
> from #991967, at which point I will report the bug I am
> seeing as a new bug.
Chuck Zmudzinski you are getting rather close to my threshold for calling
harrassment.  You're not /quite/ there, but I'm concerned.


Since the purpose of the bug reports is to find and diagnose bugs, I did
a bit of experimentation and made some observations.

I checked out the Debian Xen source via git.  I got the current
"master" branch which is presently the candidate 4.14.3-1 version,
which includes urgent fixes.  The hash is:
e7a17db0305c8de891b366ad37777528e5a43015

On top of this I cherry-picked 3 commits from Xen's main branch:
5a4087004d1adbbb223925f3306db0e5824a2bdc
0f089bbf43ecce6f27576cb548ba4341d0ec46a8
bc141e8ca56200bdd0a12e04a6ebff3c19d6c27b

(these can be retrieved via Xen's gitweb at
https://xenbits.xen.org/gitweb/?p=xen.git;a=patch;h=<$hash> which is
suitable for the `git am` command)

With these I built 4.14.3-1 and then tried kernels 4.19.181-1 and
4.19.194-3 (this system is presently mostly on oldstable).  The results
were:

Xen 4.14.3-1 with Linux 4.19.181-1: system reboots were successful

Xen 4.14.3-1 with Linux 4.19.194-3: system reboots hung

Unfortunately I was too quick at installing the rebuilt 4.14.3-1 and I
missed trying the vanilla Debian 4.14.2+25-gb6a8c4f72d-2 with
Linux 4.19.181-1.  I believe this combination would have hung during
reboot.


As such, I believe there are in fact two distinct bugs being observed.
The presence of EITHER of these is sufficient to cause hangs during
powerdown or reboot.

First, some patch originally from Linux's main branch breaks Xen reboots
was backported somewhere between 4.19.181-1 and 4.19.194-3.  This may
either have been introduced before 5.10 diverged from main, or may also
have been backported to 5.10.  THIS is Debian bug #991967.

Second, the Xen patch 3c428e9ecb1f290689080c11e0c37b793425bef1 which is
valuable to ARM devices breaks reboots and powerdowns on x86.  This is
correctly fixed by 0f089bbf43ecce6f27576cb548ba4341d0ec46a8.  Presently
this has no Debian bug report.


The first is presently unidentified, someone enthusiastic either needs to
read git logs/source code, or bisect and build to find where it got
broken.

The second we seem to have a fix.  The only question is how many patches
to cherry pick?  bc141e8ca562 is non-urgent as it is merely superficial
and not needed for functionality.
5a4087004d1a is a workaround for Linux kernel breakage, but how likely
are we to see that fixed in the Linux kernel packages?  The fix is
well-contained and needed for some highly popular ARM devices.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg at m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Hi Elliott,

On zondag 26 september 2021 05:27:07 CEST Elliott Mitchell
wrote:
> I checked out the Debian Xen source via git.  I got the current
> "master" branch which is presently the candidate 4.14.3-1
version,
> which includes urgent fixes.  The hash is:
> e7a17db0305c8de891b366ad37777528e5a43015
> 
> On top of this I cherry-picked 3 commits from Xen's main branch:
> 5a4087004d1adbbb223925f3306db0e5824a2bdc
> 0f089bbf43ecce6f27576cb548ba4341d0ec46a8
> bc141e8ca56200bdd0a12e04a6ebff3c19d6c27b
Shutdown on my Xen server broke for me between 4.14.0+80-gd101b417b7-1 and 
4.14.0+88-g1d1d1f5391-1 (too) and 'Knorrie' and I have been doing some 
experiments. We identified the 0f089bbf43 commit too, but also 2 other ones:

8b6d55c1261820bb9db8d867ce9ee77397d05203
f390941a92f102ebbbbce1b54be206a602187fd7

https://salsa.debian.org/xen-team/debian-xen/-/commits/knorrie/for-diederik-3-fixes/
is a branch Knorrie prepared for me with those 3 patches applied.
I did 'git checkout' on that branch and then a 'dpkg-buildpackage
-b' and
installed the built .deb files and rebooted. After that, shutdown worked again
:)

So you may want to take a look at those patches too.

HTH,
  Diederik
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: This is a digitally signed message part.
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20210926/59fc7860/attachment.sig>

On 9/25/2021 11:27 PM, Elliott Mitchell wrote:
>
> Unfortunately I was too quick at installing the rebuilt 4.14.3-1 and I
> missed trying the vanilla Debian 4.14.2+25-gb6a8c4f72d-2 with
> Linux 4.19.181-1.  I believe this combination would have hung during
> reboot.
>
>
In light of what I discovered while investigating the cause of
bug #994899, I would tend to think calling
Debian 4.14.2+25-gb6a8c4f72d-2 "vanilla" an interesting
choice of words. To me, vanilla connotes boring,
uninteresting. But that version of Debian Xen, and
also the current version in the stable distribution,
bullseye, are not boring or uninteresting as I have
studied these versions and concluded they actually
are now a fork of upstream Xen's 4.14 version, since
they contain patches from upstream Xen's 4.16 unstable
branch to better support the Raspberry Pi 4, as noted
in the changelogs of those versions.

So I am adding the tag upstream, and I suggest that
the Debian Xen Team notify upstream Xen that we
are planning a fork of Xen to better support popular
arm devices and we are already shipping a testing
version of it in our current bullseye release. We could
tell upstream we are willing to stop this fork if they
could assist us with backporting the reworking of the
xen/arm/acpi and xen/x86/acpi code that is in upstream
Xen 4.16 unstable to xen 4.14. We can tell
them if they are interested in what we are doing, they
can take a look at the work we are doing on our
public development servers (salsa).

For our own users, especially in the stable version,
we should make a note of this fact in a README.Debian
file and place it in an appropriate place of the binary
packages. We should also note that there are encouraging
results with this version for improved support on arm,
but some tests indicate an annoying bug causing
problems shutting down Domain 0 appear to have
surfaced on x86 (amd64). For details, see bugs #991967
and #994899 on the Debian Bug Tracking System.

I think this is the BEST way to truly proceed in accordance
with the Debian Social Policy of courtesy and cooperation
with the free software projects that are available to the
public in our main repositories, and to properly inform
our users what we are doing in our current Xen packages
for unstable, testing, and stable.

On 9/25/2021 11:27 PM, Elliott Mitchell wrote:
>
> I checked out the Debian Xen source via git.  I got the current
> "master" branch which is presently the candidate 4.14.3-1
version,
> which includes urgent fixes.  The hash is:
> e7a17db0305c8de891b366ad37777528e5a43015
>
> On top of this I cherry-picked 3 commits from Xen's main branch:
> 5a4087004d1adbbb223925f3306db0e5824a2bdc
> 0f089bbf43ecce6f27576cb548ba4341d0ec46a8
> bc141e8ca56200bdd0a12e04a6ebff3c19d6c27b
>
> (these can be retrieved via Xen's gitweb at
> https://xenbits.xen.org/gitweb/?p=xen.git;a=patch;h=<$hash> which is
> suitable for the `git am` command)
>
> With these I built 4.14.3-1 and then tried kernels 4.19.181-1 and
> 4.19.194-3 (this system is presently mostly on oldstable).  The results
> were:
>
> Xen 4.14.3-1 with Linux 4.19.181-1: system reboots were successful
>
> Xen 4.14.3-1 with Linux 4.19.194-3: system reboots hung
I presume the Xen 4.14.3-1 you are referring to is not the
official version, but the one patched with the three extra
aforementioned commits. Note: I use quilt to manage the
packages, and quilt rejected the last commit because the
context within three lines of the patched code was changed.
A goto bad was changed to goto done by another commit
on the Xen unstable branch, so I fixed the patch file
and changed the 'done' to 'bad' to get the third patch to
succeed.
Let's call this patched version of Xen version 4.14.3-1.1

I tried these on my hardware, which is a Haswell processor, EFI
boot, and systemd for init, and my results are:

Xen 4.14.3-1.1 with Linux 4.19.181-1: system reboots hung
Xen 4.14.3-1.1 with Linux 4.19.194-3: system reboots hung
Xen 4.14.3-1.1 with Linux 5.10.46-4: system reboots hung

I still cannot reproduce this result, not even with the extra three
commits. Perhaps it depends on differences in the BIOS or EFI, or
maybe systemd vs. sysv.

I share this result in case it is of help to you.

Regards,

Chuck Zmudzinski