Pkg xen devel - Oct 2021 - Bug#988333: linux-image-5.10.0-6-amd64: VGA Intel IGD Passthrough to Debian Xen HVM DomUs not working, but Windows Xen HVMs do work

Hi!

On 10/19/21 5:44 AM, Chuck Zmudzinski wrote:
> On 5/10/2021 1:33 PM, Chuck Zmudzinski wrote:
>> [...] with buster and bullseye running as the Dom0, I can only get the
VGA/Passthrough feature to work with Windows Xen HVMs. I would expect both
Windows and Linux HVMs to work comparably well.
You don't mention the used Xen version (Debian package version) for
buster and bullseye anywhere, so I'll assume it's the latest
4.14.3-1(~deb11u1) one.
> [...]
> 
> The biggest problems were that the Dom0 reported problems
> with IRQ 16 being disabled after starting the bullseye HVM DomU,
> and only xl destroy could be used to stop the corrupted process.
Well, at least we have an error somewhere already. That's a starting point.

Can you share the domU config file?

And, other configs you need to have in place to exclude the devices from
being seen as normal devices directly in dom0? (I haven't used
passthrough myself yet, but I read that this is needed.)

Can you share more verbose logging done by xl create when using xl -vvv
create <configfile>?

But, AFAIK what you want to do should be possible yes.
> The bullseye HVM DomU still fails to boot on an up-to-date
> bullseye Xen Dom0 configured to pass through the same PCI/IGD
> devices. The bullseye HVM DomU with IGD passthrough has so
> far only been verified to work on an old, slightly modified
> jessie Xen Dom0.
> 
> More Details: These latest tests are with linux version 5.10.70-1
> for bullseye stable. For the jessie Dom0, which worked with the
> unmodified bullseye HVM DomU, I had to add a few patches to
> the old jessie Xen packages so the unmodified bullseye Xen HVM
Ok, yes, clear, that makes the domU kernel not the primary suspect.
> These tests demonstrate that a fix for this bug is possible in src:xen
> rather than in src:linux, but the patches needed to fix this bug in
> Xen 4.14, which is the version of Xen on bullseye, are not yet
> identified.
It might also be possible (just a wild guess) that for Xen 4.14, the
options in the domU config file need to be different than for Xen 4.4.
> I will continue to investigate this issue and try to bisect the problem
> as it recurs in Dom0 for some version of Xen > 4.4 and <= 4.14. It
> will obviously take some time since there are so many differences
> between Xen 4.4 and 4.14.
If you can make progress on that, and find an actual commit that changes
the behavior, then we're probably at 95% towards finding a cause and
solution. :) That'd be great.

A possible time-saver that I can recommend is to send a post to the
upstream xen-users list [0] about this already. Like "Hi all, I'm
starting a HVM Linux domU with Linux 5.10.70 on a Xen 4.14.3 system with
also 5.10.70 dom0 kernel, with this and this domU config file. It fails
to start, this is the xl -vvv create output, and this error (the irq
stuff) appears in the dom0 kernel log.". Try to keep it simple and not
too long initially, without the surrounding stories, to increase chance
of it being fully read.
> If I find a fix in src:xen for Xen >=4.14 Dom0 on bullseye or sid, I
will
> reassign #988333 to src:xen myself. Until then, I will leave it to the
> discretion of the Debian Kernel Team to decide whether or not to
> reassign it to src:xen now.
Yes, that makes sense indeed, I'll do it in a minute. Even while we
don't know if it has to do with the Xen or dom0 kernel code, it's more
likely that in either case, we'll end up asking the upstream Xen people
about it.

Have fun,
Hans

[0] https://lists.xenproject.org/mailman/listinfo/xen-users

On 10/23/2021 11:11 AM, Hans van Kranenburg wrote:
> Hi!
>
> On 10/19/21 5:44 AM, Chuck Zmudzinski wrote:
>> On 5/10/2021 1:33 PM, Chuck Zmudzinski wrote:
>>> [...] with buster and bullseye running as the Dom0, I can only get
the VGA/Passthrough feature to work with Windows Xen HVMs. I would expect both
Windows and Linux HVMs to work comparably well.
> You don't mention the used Xen version (Debian package version) for
> buster and bullseye anywhere, so I'll assume it's the latest
> 4.14.3-1(~deb11u1) one.
Yes, That's the version. The original report from five months ago was an 
earlier version
but the latest version still behaves the same way. I just tested it a 
couple of days
ago.
>
>> [...]
>>
>> The biggest problems were that the Dom0 reported problems
>> with IRQ 16 being disabled after starting the bullseye HVM DomU,
>> and only xl destroy could be used to stop the corrupted process.
> Well, at least we have an error somewhere already. That's a starting
point.
>
> Can you share the domU config file?
Yes, here it is:

builder = 'hvm'
memory = '3072'
vcpus = '4'
device_model_version = 'qemu-xen'
# device_model_version = 'qemu-xen-traditional'
# This is now bullseye
disk =
['/dev/systems/linux,,xvda,w','/dev/data/linuxdata,,xvdb,w']
name = 'bullseye-hvm'
vif = [ 
'mac=00:16:3E:27:2C:AA,model=e1000,script=vif-route.hvm,ip=192.168.1.4'
]
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
boot = 'c'
acpi = '1'
apic = '1'
viridian = '1'
xen_platform_pci = '1'
serial = 'pty'
vga = 'none'
sdl = '0'
vnc = '0'
gfx_passthru = '1'
pci = [ '00:1b.0', '00:14.0,rdm_policy=relaxed',
'00:02.0' ]
>
> And, other configs you need to have in place to exclude the devices from
> being seen as normal devices directly in dom0? (I haven't used
> passthrough myself yet, but I read that this is needed.)
I run this script in Dom0 before starting the domain:

#!/bin/bash
modprobe xen-pciback
xl pci-assignable-add 00:02.0
xl pci-assignable-add 00:14.0
xl pci-assignable-add 00:1b.0
xl pci-assignable-list

The script makes the Intel IGD, USB 3.0 controller, and
sound device available to the domain. the pci = ...
statement in the domain config corresponds to these same
three PCI devices.
>
> Can you share more verbose logging done by xl create when using xl -vvv
> create <configfile>?
I don' have time now, but will do this and report tomorrow.
>
> But, AFAIK what you want to do should be possible yes.
>
>> The bullseye HVM DomU still fails to boot on an up-to-date
>> bullseye Xen Dom0 configured to pass through the same PCI/IGD
>> devices. The bullseye HVM DomU with IGD passthrough has so
>> far only been verified to work on an old, slightly modified
>> jessie Xen Dom0.
>>
>> More Details: These latest tests are with linux version 5.10.70-1
>> for bullseye stable. For the jessie Dom0, which worked with the
>> unmodified bullseye HVM DomU, I had to add a few patches to
>> the old jessie Xen packages so the unmodified bullseye Xen HVM
> Ok, yes, clear, that makes the domU kernel not the primary suspect.
>
>> These tests demonstrate that a fix for this bug is possible in src:xen
>> rather than in src:linux, but the patches needed to fix this bug in
>> Xen 4.14, which is the version of Xen on bullseye, are not yet
>> identified.
> It might also be possible (just a wild guess) that for Xen 4.14, the
> options in the domU config file need to be different than for Xen 4.4.
They are a little different already, 4.4 did not need the rdm_policy 
setting.
But you are right, there are other settings I haven't checked yet. I 
will report
on some more tests I have done tomorrow when I have more ti
>
>> I will continue to investigate this issue and try to bisect the problem
>> as it recurs in Dom0 for some version of Xen > 4.4 and <= 4.14.
It
>> will obviously take some time since there are so many differences
>> between Xen 4.4 and 4.14.
> If you can make progress on that, and find an actual commit that changes
> the behavior, then we're probably at 95% towards finding a cause and
> solution. :) That'd be great.
>
> A possible time-saver that I can recommend is to send a post to the
> upstream xen-users list [0] about this already. Like "Hi all, I'm
> starting a HVM Linux domU with Linux 5.10.70 on a Xen 4.14.3 system with
> also 5.10.70 dom0 kernel, with this and this domU config file. It fails
> to start, this is the xl -vvv create output, and this error (the irq
> stuff) appears in the dom0 kernel log.". Try to keep it simple and not
> too long initially, without the surrounding stories, to increase chance
> of it being fully read.
I can do this soon - I have some more interesting tests to share
here and with the Xen developers upstream.
>
>> If I find a fix in src:xen for Xen >=4.14 Dom0 on bullseye or sid, I
will
>> reassign #988333 to src:xen myself. Until then, I will leave it to the
>> discretion of the Debian Kernel Team to decide whether or not to
>> reassign it to src:xen now.
> Yes, that makes sense indeed, I'll do it in a minute. Even while we
> don't know if it has to do with the Xen or dom0 kernel code, it's
more
> likely that in either case, we'll end up asking the upstream Xen people
> about it.
>
> Have fun,
OK, I will!

Cheers,

Chuck