Pkg xen devel - Oct 2021 - linux-image-5.10.0-6-amd64: VGA Intel IGD Passthrough to Debian Xen HVM DomUs not working, but Windows Xen HVMs do work

On 5/10/2021 1:33 PM, Chuck Zmudzinski wrote:
> Package: src:linux
> Version: 5.10.28-1
> Severity: normal
> Tags: upstream
>
> Dear Maintainer,
>
> I have been using Xen's PCI and VGA passthrough feature since wheezy
and jessie were the stable versions, and back then both Windows HVMs and Linux
HVMs would function with the Intel Integrated Graphics Device (IGD), the audio
device, and the USB 3 controller passed to them. But with buster and bullseye
running as the Dom0, I can only get the VGA/Passthrough feature to work with
Windows Xen HVMs. I would expect both Windows and Linux HVMs to work comparably
well.
>
>
Dear Debian Kernel Team and Debian Xen Team,

I originally reported this bug in src:linux, as described above, but
recent tests indicate a fix can be made in src:xen without any
modifications to src:linux, so I suggest reassigning it from
src:linux to src:xen. My explanation follows:

On my system which is an ASRock B85M Pro4 (Haswell), with BIOS
P2.50 12/11/2015, and with a jessie Xen Dom0 with a few patches
to the old jessie Xen packages, I was able to successfully pass
through the USB 3.0 controller, the sound card, and the Intel IGD
to an unmodified bullseye HVM DomU without any of the
problems I reported in the original bug report (message #5):

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988333#5

The biggest problems were that the Dom0 reported problems
with IRQ 16 being disabled after starting the bullseye HVM DomU,
and only xl destroy could be used to stop the corrupted process.

The bullseye HVM DomU still fails to boot on an up-to-date
bullseye Xen Dom0 configured to pass through the same PCI/IGD
devices. The bullseye HVM DomU with IGD passthrough has so
far only been verified to work on an old, slightly modified
jessie Xen Dom0.

More Details: These latest tests are with linux version 5.10.70-1
for bullseye stable. For the jessie Dom0, which worked with the
unmodified bullseye HVM DomU, I had to add a few patches to
the old jessie Xen packages so the unmodified bullseye Xen HVM
DomU would boot on the jessie Xen Dom0 that uses a fairly
old version of Xen (version 4.4). Specifically, it was necessary to
add two upstream Xen patches to the old jessie Xen-4.4 packages:

1. https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=98297f0
2. https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=09a4ef8

The first patch is needed to support booting Linux kernels >= 4.10 in
a Xen HVM DomU on a Xen 4.4 Dom0, since that is when the Linux
kernel started validating the timestamp counter adjust msr for hvm
guests, and the validation fails on Xen 4.4 without the first patch.

Modern versions of Linux expect the Dom0 to provide feature-XXX
flags in xenstore for the DomU. The 4.4 version of libxl does not
provide this so the second patch provides it for version 4.4 of libxl.

Neither of these two patches specifically solves the problem with
IGD passthrough, they are simply needed to enable the old Xen 4.4
hypervisor and tools to boot modern Linux kernels in a Xen HVM
DomU, with or without PCI/IGD passthrough.

It was also necessary to use the ancient Xen qemu traditional device
model since the old Xen 4.4 does not support IGD passthrough with
the older upstream qemu device model for jessie. I did not do
anything special here. I just compiled the qemu-xen-traditional
binary from the xenbits git repository:

https://xenbits.xen.org/gitweb/?p=qemu-xen-traditional.git;a=commit;h=204a7fc1

and recompiled hvmloader with rombios as required by
qemu-xen-traditional and integrated these with the Debian
packaging of Xen 4.4 for jessie the way it was done with
Xen 4.1 on wheezy before Debian removed qemu-xen-traditional
from the Debian Xen packages.

On the jessie Dom0, no other changes were made. For example, it
used the latest 3.16.0-11 kernel for jessie.

These tests demonstrate that a fix for this bug is possible in src:xen
rather than in src:linux, but the patches needed to fix this bug in
Xen 4.14, which is the version of Xen on bullseye, are not yet
identified.

I will continue to investigate this issue and try to bisect the problem
as it recurs in Dom0 for some version of Xen > 4.4 and <= 4.14. It
will obviously take some time since there are so many differences
between Xen 4.4 and 4.14.

If I find a fix in src:xen for Xen >=4.14 Dom0 on bullseye or sid, I will
reassign #988333 to src:xen myself. Until then, I will leave it to the
discretion of the Debian Kernel Team to decide whether or not to
reassign it to src:xen now.

Regards,

Chuck

Hi!

On 10/19/21 5:44 AM, Chuck Zmudzinski wrote:
> On 5/10/2021 1:33 PM, Chuck Zmudzinski wrote:
>> [...] with buster and bullseye running as the Dom0, I can only get the
VGA/Passthrough feature to work with Windows Xen HVMs. I would expect both
Windows and Linux HVMs to work comparably well.
You don't mention the used Xen version (Debian package version) for
buster and bullseye anywhere, so I'll assume it's the latest
4.14.3-1(~deb11u1) one.
> [...]
> 
> The biggest problems were that the Dom0 reported problems
> with IRQ 16 being disabled after starting the bullseye HVM DomU,
> and only xl destroy could be used to stop the corrupted process.
Well, at least we have an error somewhere already. That's a starting point.

Can you share the domU config file?

And, other configs you need to have in place to exclude the devices from
being seen as normal devices directly in dom0? (I haven't used
passthrough myself yet, but I read that this is needed.)

Can you share more verbose logging done by xl create when using xl -vvv
create <configfile>?

But, AFAIK what you want to do should be possible yes.
> The bullseye HVM DomU still fails to boot on an up-to-date
> bullseye Xen Dom0 configured to pass through the same PCI/IGD
> devices. The bullseye HVM DomU with IGD passthrough has so
> far only been verified to work on an old, slightly modified
> jessie Xen Dom0.
> 
> More Details: These latest tests are with linux version 5.10.70-1
> for bullseye stable. For the jessie Dom0, which worked with the
> unmodified bullseye HVM DomU, I had to add a few patches to
> the old jessie Xen packages so the unmodified bullseye Xen HVM
Ok, yes, clear, that makes the domU kernel not the primary suspect.
> These tests demonstrate that a fix for this bug is possible in src:xen
> rather than in src:linux, but the patches needed to fix this bug in
> Xen 4.14, which is the version of Xen on bullseye, are not yet
> identified.
It might also be possible (just a wild guess) that for Xen 4.14, the
options in the domU config file need to be different than for Xen 4.4.
> I will continue to investigate this issue and try to bisect the problem
> as it recurs in Dom0 for some version of Xen > 4.4 and <= 4.14. It
> will obviously take some time since there are so many differences
> between Xen 4.4 and 4.14.
If you can make progress on that, and find an actual commit that changes
the behavior, then we're probably at 95% towards finding a cause and
solution. :) That'd be great.

A possible time-saver that I can recommend is to send a post to the
upstream xen-users list [0] about this already. Like "Hi all, I'm
starting a HVM Linux domU with Linux 5.10.70 on a Xen 4.14.3 system with
also 5.10.70 dom0 kernel, with this and this domU config file. It fails
to start, this is the xl -vvv create output, and this error (the irq
stuff) appears in the dom0 kernel log.". Try to keep it simple and not
too long initially, without the surrounding stories, to increase chance
of it being fully read.
> If I find a fix in src:xen for Xen >=4.14 Dom0 on bullseye or sid, I
will
> reassign #988333 to src:xen myself. Until then, I will leave it to the
> discretion of the Debian Kernel Team to decide whether or not to
> reassign it to src:xen now.
Yes, that makes sense indeed, I'll do it in a minute. Even while we
don't know if it has to do with the Xen or dom0 kernel code, it's more
likely that in either case, we'll end up asking the upstream Xen people
about it.

Have fun,
Hans

[0] https://lists.xenproject.org/mailman/listinfo/xen-users