CentOS - Jul 2020 - 8.2.2004 Latest yum update renders machine unbootable

I am running an Intel x64 machine using UEFI to boot an SSD.

Installing the latest yum update which includes grub2 and kernel 
4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank 
screen where grub should be, no error messages, just hangs.

After some hours I managed to modify another bootable partition 
(containing older software) and boot it from there.

After that, I? found out it is a known problem.

The main point of this message is to make people aware of the problem 
and suggest admins don't run 'yum update' until they understand the 
problem and have a fix at hand.

See 'UEFI boot blank screen post update' for a solution and directions 
to the redhat article.

Regards

Alan

On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS
wrote:
> I am running an Intel x64 machine using UEFI to boot an SSD.
> 
> Installing the latest yum update which includes grub2 and kernel 
> 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank 
> screen where grub should be, no error messages, just hangs.
> 
> After some hours I managed to modify another bootable partition 
> (containing older software) and boot it from there.
> 
> After that, I  found out it is a known problem.
> 
> The main point of this message is to make people aware of the problem 
> and suggest admins don't run 'yum update' until they understand
the
> problem and have a fix at hand.
> 
> See 'UEFI boot blank screen post update' for a solution and
directions
> to the redhat article.
> 
> Regards
> 
> Alan
> 
I have been punished by this bug - it is/was very nasty.

I will chase up your reference but attach my trouble shooting
in case its useful to anyone

John
-------------- next part --------------
31.07.2020

Tried, successfully, to fix the boot problems on maui (Centos 7.8)

To get maui to boot at all an F32 USB stick was used
and the first F32 boot option edited for every boot attempt.

The changes required are summarised below.
insmod part_gpt
insmod ext2
linux (hd2, gpt3)/vm ... 18 ... root=/dev/sda5
iinitrd (hd2, gpt3)/initramfs ... 18 ...

The above booted OK with a 90 second delay if /global & /home SSDs were not
plugged in.
fstab was subsequently edited and the nfs related entries temporarily commented
out.
//-------------------------------------------------------------------------------------------
The following rpms were reinstalled - it is not known if this was necessary -
probably not.
grub2-efi-x64
efibootmgr
efivar-libs
fwupdate-efi
shim-x64

grub.cfg was re-created but has never been used.
grub2-mkconfig -o ~/grub.cfg_post_2020_07_30

cp /boot/efi/EFI/centos/grub.cfg /boot/efi/EFI/centos/grub.cfg_pre_2020_07_30

The old grub.cfg has not been over written yet as the boot does not get that
far!
//-------------------------------------------------------------------------------------------
The use of efibootmgr was investigated and a corrupt looking entry removed.
A new entry was also created.

efibootmgr -v						To list entries
efiboot mgr -b 2 -B 2				To remove entry 2 say
efibootmgr -t 10					To increase time to edit
efibootmgr -c -d /dev/sda -p1 -L ja_C7 -l '\EFI\centos\grubx64.efi'

-c 							create new entry
-d							Device
-p							vfat partition number
-L							Label
-l							boot loader

The choice for the "boot loader" option would seem to be
shimx64.efi
shimx64-centos.efi
shim.efi
grubx64.efi
Which version is correct is unknown, but '\EFI\centos\grubx64.efi' was
the first tried and works!

The resultant entries look like this:

[root at maui:/boot/efi/EFI/centos]$ efibootmgr -v
BootCurrent: 0006
Timeout: 10 seconds
BootOrder: 0002,0001,0000,0005,0006
Boot0000* Fedora       
HD(2,GPT,7c336c1e-9b29-4297-9b6f-ad7afc651a43,0x1000,0x200000)/File(\EFI\FEDORA\SHIMX64.EFI)
Boot0001* Fedora       
HD(2,GPT,7c336c1e-9b29-4297-9b6f-ad7afc651a43,0x1000,0x200000)/File(\EFI\FEDORA\shimx64.efi)
Boot0002* ja_C7
HD(1,GPT,751f986e-faf4-407e-9db0-caef0c8cd546,0x800,0x64000)/File(\EFI\centos\grubx64.efi)
Boot0005* UEFI OS      
HD(1,GPT,751f986e-faf4-407e-9db0-caef0c8cd546,0x800,0x64000)/File(\EFI\BOOT\BOOTX64.EFI)..BO
Boot0006* UEFI OS      
HD(2,GPT,7c336c1e-9b29-4297-9b6f-ad7afc651a43,0x1000,0x200000)/File(\EFI\BOOT\BOOTX64.EFI)..BO

Conclusion

The above restored maui to a normal boot state.
It is probable that the "EFI BIOS" became corrupted somehow.
The file \EFI\BOOT\BOOTX64.EFI in menu entry 5 does exist but this option has
not been tested yet!

Il 31/07/20 13:08, ja ha scritto:
> On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
>> I am running an Intel x64 machine using UEFI to boot an SSD.
>>
>> Installing the latest yum update which includes grub2 and kernel
>> 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank
>> screen where grub should be, no error messages, just hangs.
>>
>> After some hours I managed to modify another bootable partition
>> (containing older software) and boot it from there.
>>
>> After that, I  found out it is a known problem.
>>
>> The main point of this message is to make people aware of the problem
>> and suggest admins don't run 'yum update' until they
understand the
>> problem and have a fix at hand.
>>
>> See 'UEFI boot blank screen post update' for a solution and
directions
>> to the redhat article.
>>
>> Regards
>>
>> Alan
>>
> I have been punished by this bug - it is/was very nasty.
>
Me too. Luckily it happened on a test machine.

Sorry but seems that those packages were not tested before pushing them 
in the update repo. Would be great to know what happened to the 
mainstream chains and how a package like grub reached the update repo 
when it has serious problem (genuine curiosity but not to blame them).

In my case, the restore procedure reported by RH in case of reboot does 
not work and reports that the packages are already at the lowest version 
and that the downgrade is not possibile. I don't know why.

On 7/31/20 6:35 AM, Alan McRae via CentOS wrote:
> I am running an Intel x64 machine using UEFI to boot an SSD.
>
> Installing the latest yum update which includes grub2 and kernel 
> 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank 
> screen where grub should be, no error messages, just hangs.
>
> ...
> The main point of this message is to make people aware of the problem 
> and suggest admins don't run 'yum update' until they understand
the
> problem and have a fix at hand. 
I posted, on 7/30, to CentOS-devel the following after my similar 
experience (booting an mSATA SSD on my M6700, with two 2.5 SATA drives 
(a 1TB and a 2TB), and the mSATA is not /dev/sda on this hardware, FWIW):

"Moral of the story?? Have the install DVD on a USB stick and available 
to boot into rescue mode, know how to boot rescue mode (from the 
installer boot menu select 'troubleshooting....' then
'rescue....' and
option 1 once the menu shows), know how to activate network interfaces 
in text mode (either nmcli or nmtui works fine for this), and know a few 
basic dnf commands.? And have an alternate means of doing basic web 
searches available; my android phone this morning was used for that....."

I was able to recover with the install DVD on a USB stick and using 
rescue mode on a laptop with only WiFi (nmtui does a great job for this 
sort of thing, booted into rescue mode and chrooted onto /mnt/sysimage).