CentOS - Jan 2018 - CentOS 7.4 fails to boot as Xen PV guest: resurfaces (now also) with centosplus kernel 693.11.6.el7

Dear all,

Maybe I'm the only one - so before filing it as a bug: it appears that
the latest set of kernel patches in 3.10.0-693.11.6.el7 makes issue
0013763 "CentOS 7.4 kernel (3.10.0-693*) fails to boot as Xen PV
guest"
re-surface *also* with the CentOS PLUS kernel. But maybe in a
different way ...

Thanks to the (great!) quick work on making the plus kernel available
(in #14330, thanks for that!) I was able to test the following combinations:

- 3.10.0-693.11.6.el7.centos.plus works fine on XenServer 7 HVM guests
- 3.10.0-693.11.1.el7.centos.plus works fine on XenServer 7 XenPV guests
- 3.10.0-693.11.6.el7.centos.plus crashes on boot for XS7 XenPV guests
  immediately after the boot stage where the PS/2 drivers are loaded

Crashing does not depend on the amount of VM memory or #cores/sockets.
Also, the tell-tale "mm/vmalloc.c" failure output is no longer
present.
The system 'just' crashes (with the kernel log lines as below) around
the
time of the i8042 controller probing.

Obviously the stock upstream 3.10.0-693.11.6.el7 crashes as well for
XS7 XenPV guests, but that was to be expected as per 0013763.
It crashes in the same way, around the time of the i8042 probes.

Can anyone (maybe bill_mcgonigle?) reproduce the issue?
Did the KPTI patches break the XenPV fixes in CentOS Plus per 13763?
Or is this a new XenPV issue?

Interestingly, the crash is now sudden and (even at loglevel=7)
without further information. The crash after the i8042/mousedev
is (I think) unrelated to these devices, since the system crashes in the
same way even with "i8042.debug i8042.nokbd i8042.noaux i8042.dumbkbd
i8042.nomux i8042.noloop i8042.nopnp psmouse.proto=bare" added to the
kernel command line (and then you don't see any PS/2 and mousedev).
So something immediately after the mousy things triggers the crash:

 0.2293541 io scheduler cfq registered
 0.2295071 pci_hotplug: PCI Hot Plug PCI Core version: 0.5
 0.2295241 pciehp: PCI Express Hot Plug Controller Driver version: 0.4
 0.2301241 Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
 0.2306111 Non-volatile memory driver u1.3
 0.2306601 Linux agpgart interface u0.103
 0.2307641 crash memory driver: version 1.1
 0.2308301 rdac: device handler registered
 0.2308741 hp_sw: device handler registered
 0.2308851 emc: device handler registered
 0.2308941 alua: device handler registered
 0.2309491 libphy: Fixed MDIO Bus: probed
 0.2310301 ehci_hcd: USB 2.0 'Enhanced' Host Controller CEHCI) Driver
 0.2310471 ehci-pci: EHCI PCI platform driver
 0.2310671 ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
 0.2310791 ohci-pci: OHCI PCI platform driver
 0.2310951 uhci_hcd: USB Universal Host Controller Interface driver
 0.2311581 usbcore: registered new interface driver usbserial
 0.2311741 usbcore: registered new interface driver usbserial_generic
 0.2311941 usbserial: USB Serial support registered for generic
 0.2312291 i8042: PNP: No PS/2 controller found. Probing ports directly.
 1.2427071 i8042: No controller found
 1.2428571 mousedev: PS/2 mouse device common for all mice
 [HANG - VM shuts down]

although sometimes the boot stops at "i8042: PNP: No PS/2 controller
found. Probing ports directly." and does not make it to the
"mouse" bit.

Anyone found similar behaviour?

	Thanks a lot!
	DavidG.

-- 
David Groep

** Nikhef, Dutch National Institute for Subatomic Physics,  PDP programme **
** Room: H1.50 Phone: +31 20 5922179, PObox 41882, NL-1009DB Amsterdam NL **
** PGP: 0xD80134C2 308E076A  FP: 2facebea12803ba145685a21d80134c2308e076a **

On Sun, Jan 7, 2018 at 12:07 PM, David Groep <davidg at nikhef.nl> wrote:
> Dear all,
>
> Maybe I'm the only one - so before filing it as a bug: it appears that
> the latest set of kernel patches in 3.10.0-693.11.6.el7 makes issue
> 0013763 "CentOS 7.4 kernel (3.10.0-693*) fails to boot as Xen PV
guest"
> re-surface *also* with the CentOS PLUS kernel. But maybe in a
> different way ...
>
> Thanks to the (great!) quick work on making the plus kernel available
> (in #14330, thanks for that!) I was able to test the following
> combinations:
>
> - 3.10.0-693.11.6.el7.centos.plus works fine on XenServer 7 HVM guests
> - 3.10.0-693.11.1.el7.centos.plus works fine on XenServer 7 XenPV guests
> - 3.10.0-693.11.6.el7.centos.plus crashes on boot for XS7 XenPV guests
>   immediately after the boot stage where the PS/2 drivers are loaded
>
> Crashing does not depend on the amount of VM memory or #cores/sockets.
> Also, the tell-tale "mm/vmalloc.c" failure output is no longer
present.
> The system 'just' crashes (with the kernel log lines as below)
around the
> time of the i8042 controller probing.
>
> Obviously the stock upstream 3.10.0-693.11.6.el7 crashes as well for
> XS7 XenPV guests, but that was to be expected as per 0013763.
> It crashes in the same way, around the time of the i8042 probes.
>
> Can anyone (maybe bill_mcgonigle?) reproduce the issue?
> Did the KPTI patches break the XenPV fixes in CentOS Plus per 13763?
> Or is this a new XenPV issue?
>
?You may want to be watching the centos-virt mailing list. Xen-related
issues are discussed there.

This thread:

?
https://lists.centos.org/pipermail/centos-virt/2018-January/005716.html

?has a response from Johnny Hughes with a possible solution.

Akemi?

On 01/08/2018 12:38 AM, Akemi Yagi wrote:
> On Sun, Jan 7, 2018 at 12:07 PM, David Groep <davidg at nikhef.nl>
wrote:
> 
>> Dear all,
>>
>> Maybe I'm the only one - so before filing it as a bug: it appears
that
>> the latest set of kernel patches in 3.10.0-693.11.6.el7 makes issue
>> 0013763 "CentOS 7.4 kernel (3.10.0-693*) fails to boot as Xen PV
guest"
>> re-surface *also* with the CentOS PLUS kernel. But maybe in a
>> different way ...
>>
>> Thanks to the (great!) quick work on making the plus kernel available
>> (in #14330, thanks for that!) I was able to test the following
>> combinations:
>>
>> - 3.10.0-693.11.6.el7.centos.plus works fine on XenServer 7 HVM guests
>> - 3.10.0-693.11.1.el7.centos.plus works fine on XenServer 7 XenPV
guests
>> - 3.10.0-693.11.6.el7.centos.plus crashes on boot for XS7 XenPV guests
>>   immediately after the boot stage where the PS/2 drivers are loaded
>>
>> Crashing does not depend on the amount of VM memory or #cores/sockets.
>> Also, the tell-tale "mm/vmalloc.c" failure output is no
longer present.
>> The system 'just' crashes (with the kernel log lines as below)
around the
>> time of the i8042 controller probing.
>>
>> Obviously the stock upstream 3.10.0-693.11.6.el7 crashes as well for
>> XS7 XenPV guests, but that was to be expected as per 0013763.
>> It crashes in the same way, around the time of the i8042 probes.
>>
>> Can anyone (maybe bill_mcgonigle?) reproduce the issue?
>> Did the KPTI patches break the XenPV fixes in CentOS Plus per 13763?
>> Or is this a new XenPV issue?
>>
> 
> ?You may want to be watching the centos-virt mailing list. Xen-related
> issues are discussed there.
> 
> This thread:
> 
> ?
> https://lists.centos.org/pipermail/centos-virt/2018-January/005716.html
> 
> ?has a response from Johnny Hughes with a possible solution.
> 
And specifically, this message has links to the 4.9.75 x86_64 kernel
that will boot and run as a Xen PV kernel:

https://lists.centos.org/pipermail/centos-virt/2018-January/005721.html

Also, the latest Xen Dom0 kernel 4.9.75 kernel will also boot a DomU Xen
PV.  That kernel is still in testing, and also requires the latest
linux-firmware rpms:

https://buildlogs.centos.org/centos/7/virt/x86_64/

or

https://buildlogs.centos.org/centos/6/virt/x86_64/


(And your xen version ... xen-44/, xen-46/, xen-48/)

Those kernels should be signed released to the main Xen repo sometime
this week.

And as someone has mentioned in another thread (Phil Perry), please read
Greg K-H's (the Kernel.org LTS maintainer, he is in charge of the 4.9.x
LTS tree that we use for the above kernels):

http://www.kroah.com/log/blog/2018/01/06/meltdown-status/

Thanks,
Johnny Hughes


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20180108/d7fcd69f/attachment-0001.sig>