Pkg xen devel - Oct 2018 - Bug#910946: xen-hypervisor-4.11-amd64: HVM DomU - 64-bit kernel cannot access emulated ATA devices

Package: xen-hypervisor-4.11-amd64
Version: 4.11.1~pre.20180911.5acdd26fdc+dfsg-3
Severity: normal

Dear Maintainer,

This mostly manifests as an "unable to access emulated CD-ROM drive",
because there are no PVHVM CD-ROMs.

NOTE: I had similar issues with 4.8, so I upgraded to 4.11. This actually gave
me *more* issues.

Attempting to boot an HVM domU from a CD image (to perform an install or rescue
on VM data) fails when the CD uses a 64-bit kernel.
(When I use the Knoppix 'failsafe' mode, which uses a 32-bit kernel, I
get different behavior.  Under 4.8, it boots; under 4.11, it hangs during ATA
bus probing.)

I have tried the following ISO images:
* debian-buster-DI-alpha3-amd64-netinst.iso (tested on 4.8 only; can't use
at all on 4.11)
* KNOPPIX_V8.2-2018-05-10-EN.iso
* debian-live-9.5.0-amd64-kde.iso
* gparted-live-0.32.0-1-amd64.iso

Here are some additional details (mostly gleaned from Knoppix, because
that's what I was doing my testing with):
- When I try booting Knoppix in a non-accelerated VM by directly running qemu,
everything works fine, so Xen is definitely a factor.
- PVHVM is working: non-CDROM drive shows up as xvbda
- With PVHVM disabled (xen_platform_pci=0), hda also fails to be detected
- Persuading libata to dump IDENTIFY data reveals that the returned page is all
zeroes.


-- System Information:
Debian Release: buster/sid
  APT prefers testing
  APT policy: (900, 'testing'), (400, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.18.6 (SMP w/12 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

xen-hypervisor-4.11-amd64 depends on no packages.

Versions of packages xen-hypervisor-4.11-amd64 recommends:
ii  xen-hypervisor-common  4.11.1~pre.20180911.5acdd26fdc+dfsg-1~exp1
ii  xen-utils-4.11         4.11.1~pre.20180911.5acdd26fdc+dfsg-3

xen-hypervisor-4.11-amd64 suggests no packages.

-- no debconf information

Hi Stephen,

On 10/13/2018 07:31 PM, Stephen Oberholtzer wrote:
> Package: xen-hypervisor-4.11-amd64
> Version: 4.11.1~pre.20180911.5acdd26fdc+dfsg-3
> Severity: normal
Thanks for trying out the new Xen 4.11 packages, spending time testing
things and writing up reports.
> Dear Maintainer,
> 
> This mostly manifests as an "unable to access emulated CD-ROM
drive", because there are no PVHVM CD-ROMs.
> 
> NOTE: I had similar issues with 4.8, so I upgraded to 4.11. This actually
gave me *more* issues.
Yeah, I've noticed that you also reported this earlier on #xen irc.

There was a reply from Andrew Cooper:
19:35 < andyhhp__> Stevie-O: Thats most likely a dom0 kernel bug
19:35 < andyhhp__> the 32 and 64bit block protocols were binarily
diverged for a while due to some "interesing" bugfixes
19:36 < andyhhp__> but its the most common cause of "32bit works,
64bit
doesn't" when it comes to anything vaguely like disks

I suspect that this issue, together with the other two issues you just
opened are not things that can easily be resolved only by the people
doing the packaging work for Debian. This means that we have to get into
the mode "let's prepare a well-written bug report for upstream xen
developers".

Since the comments from Andrew point at at least the dom0 kernel version
being a factor in this equation: can you also add linux kernel version
information for dom0 and domU for your tests? These kind of things
explode into a testing-matrix of combinations of things very fast, and
any testing you can do providing that info will help other people
helping you narrowing down the issue.

Thanks,
> Attempting to boot an HVM domU from a CD image (to perform an install or
rescue on VM data) fails when the CD uses a 64-bit kernel.
> (When I use the Knoppix 'failsafe' mode, which uses a 32-bit
kernel, I get different behavior.  Under 4.8, it boots; under 4.11, it hangs
during ATA bus probing.)
> 
> I have tried the following ISO images:
> * debian-buster-DI-alpha3-amd64-netinst.iso (tested on 4.8 only; can't
use at all on 4.11)
> * KNOPPIX_V8.2-2018-05-10-EN.iso
> * debian-live-9.5.0-amd64-kde.iso
> * gparted-live-0.32.0-1-amd64.iso
> 
> Here are some additional details (mostly gleaned from Knoppix, because
that's what I was doing my testing with):
> - When I try booting Knoppix in a non-accelerated VM by directly running
qemu, everything works fine, so Xen is definitely a factor.
> - PVHVM is working: non-CDROM drive shows up as xvbda
> - With PVHVM disabled (xen_platform_pci=0), hda also fails to be detected
> - Persuading libata to dump IDENTIFY data reveals that the returned page is
all zeroes.
> 
> 
> -- System Information:
> Debian Release: buster/sid
>   APT prefers testing
>   APT policy: (900, 'testing'), (400, 'unstable')
> Architecture: amd64 (x86_64)
> 
> Kernel: Linux 4.18.6 (SMP w/12 CPU cores)
> Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
> Init: systemd (via /run/systemd/system)
> LSM: AppArmor: enabled
> 
> xen-hypervisor-4.11-amd64 depends on no packages.
> 
> Versions of packages xen-hypervisor-4.11-amd64 recommends:
> ii  xen-hypervisor-common  4.11.1~pre.20180911.5acdd26fdc+dfsg-1~exp1
> ii  xen-utils-4.11         4.11.1~pre.20180911.5acdd26fdc+dfsg-3
> 
> xen-hypervisor-4.11-amd64 suggests no packages.
> 
> -- no debconf information
--
Hans van Kranenburg

On Sat, Oct 13, 2018 at 5:44 PM Hans van Kranenburg <hans at knorrie.org>
wrote:
> 19:35 < andyhhp__> Stevie-O: Thats most likely a dom0 kernel bug
> 19:35 < andyhhp__> the 32 and 64bit block protocols were binarily
> diverged for a while due to some "interesing" bugfixes
> 19:36 < andyhhp__> but its the most common cause of "32bit
works, 64bit
> doesn't" when it comes to anything vaguely like disks
>
>
> Since the comments from Andrew point at at least the dom0 kernel version
> being a factor in this equation: can you also add linux kernel version
> information for dom0 and domU for your tests? These kind of things
> explode into a testing-matrix of combinations of things very fast, and
> any testing you can do providing that info will help other people
> helping you narrowing down the issue.
>
>
Certainly!

The dom0 is 4.18.6.  It is a custom-built version of the
linux-image-4.18.6, with xen_pciback compiled-in (instead of being a
module.)

The domU kernels are whatever on the CD-ROMs images. The host is shut down
at the moment, but I can get those tonight or tomorrow if it's really
important.

However, I disagree with andyhhp's assessment with regard to 32-bit/64-bit
protocols: these are virtual CD-ROMs, which are not PV block devices, but
fully-emulated hardware.
They speak the ATAPI protocol, which does not change based on the bitness
of the kernel running inside the VM.
The failure is happening when the domU kernel sends an IDENTIFY DEVICE
command to find out what type of drive it is.  Since that doesn't involve
anything in dom0 (optical drives exist separately and independently from
the media inside the drives), the dom0 kernel shouldn't be getting involved
at all.

(I would very much like more information on those 'interesting bugfixes'
that affect 32/64 compatibility, however.)

-- 
-- Stevie-O
Real programmers use COPY CON PROGRAM.EXE
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20181013/d28ae00c/attachment.html>