CentOS - Mar 2023 - Kernel updates do not boot - always boots oldest kernel

This issue has been around for some months, but other things keep 
crowding out a fix.

uname give me

3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021

yet I have

3.10.0-1160.76.1.el7.x86_64
3.10.0-1160.81.1.el7.x86_64
3.10.0-1160.83.1.el7.x86_64
3.10.0-1160.88.1.el7.x86_64

loaded.

The system uses UEFI to boot.

sudo ls -l /sys/firmware/efi
gives:


total 0
-r--r--r--.? 1 root root 4096 Feb 19 16:47 config_table
drwxr-xr-x.? 2 root root??? 0 Feb 19 16:47 efivars
-r--r--r--.? 1 root root 4096 Mar 14 17:57 fw_platform_size
-r--r--r--.? 1 root root 4096 Feb 19 16:47 fw_vendor
drwxr-xr-x.? 2 root root??? 0 Mar 14 17:57 mok-variables
-r--r--r--.? 1 root root 4096 Feb 19 16:47 runtime
drwxr-xr-x.? 9 root root??? 0 Feb 19 16:47 runtime-map
-r--------.? 1 root root 4096 Feb 19 16:47 systab
drwxr-xr-x. 65 root root??? 0 Mar 14 17:57 vars

and

sudo efibootmgr

gives:

BootCurrent: 000F
BootOrder: 000F,000D,000B,000E,0008,0000,0002,0003,0004,0005,0006,0007
Boot0000* CD/DVD Rom
Boot0002* PXE Network
Boot0003? Enter Setup
Boot0004? Boot Devices
Boot0005? Boot Manager
Boot0006? Setup
Boot0007? Diagnostics
Boot0008* Embedded Hypervisor
Boot000B* CentOS Linux
Boot000D* CentOS-AltDrv
Boot000E* Hard Disk 3
Boot000F* CentOS-MainDrv

This is a remote server, thus I need a sure fire fix. My previous 
attempts have either had no impact - the old kernel boots

or

machine hangs and I need to do a trip to the site.

Now this issue could be a residual from my initial setup when I 
installed 2 by 3.x TB SSD and needed to manually change from bios/grub2 
boot to UEFI.

I have already spent 10's of hours on this system, just want to have it 
run the latest kernels - for obvious reasons.

Some other items:

sudo grep "^menuentry" /boot/grub2/grub.cfg | cut -d "'"
-f2

gives:

CentOS Linux (3.10.0-1160.88.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.83.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.81.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.76.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1160.36.2.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-a39773847cf34651bc34d02222566f53) 7 (Core)

indicating that .88.1 should boot.

sudo grub2-editenv list

gives:

saved_entry=CentOS Linux (3.10.0-1160.88.1.el7.x86_64) 7 (Core)

also as expected.

/etc/default/grub exists and contains

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g'
/etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto 
rd.md.uuid=066ffecb:69137a0b:4e579b4f:dfbf1696 
rd.md.uuid=bd87f682:e6df10e2:d2a6e247:834133f7 rhgb quiet"
GRUB_DISABLE_RECOVERY="true"

the /boot/grub2/grubenv contains

# GRUB Environment Block
saved_entry=CentOS Linux (3.10.0-1160.88.1.el7.x86_64) 7 (Core)
#######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

All these point to the correct version of the kernel but always boots 
the old .36.2 version.

Just realized these files only relate to BIOS boot, and my system is 
UEFI boot.

Now documentation seems to get scarce.

seems like the boot files are now residing in

/boot/efi/EFI/centos

AND

/boot/efi2/EFI/centos

although looking at timestamps the latter directory is not being updated.

/boot/efi/EFI/centos contains

total 7028
-rwx------. 1 root root???? 134 Aug? 1? 2020 BOOT.CSV
-rwx------. 1 root root???? 134 Aug? 1? 2020 BOOTX64.CSV
drwx------. 2 root root??? 4096 Dec 23 22:01 fonts
-rwx------. 1 root root??? 8589 Mar 14 17:51 grub.cfg
-rwx------. 1 root root??? 1024 Aug 26? 2021 grubenv
-rwx------. 1 root root 1125704 Dec 17 06:13 grubx64.efi
-rwx------. 1 root root 1154640 Aug? 1? 2020 mmx64.efi
-rwx------. 1 root root 1154640 Aug? 1? 2020 MokManager.efi
-rwx------. 1 root root 1243864 Aug? 1? 2020 shim.efi
-rwx------. 1 root root 1237824 Aug? 1? 2020 shimx64-centos.efi
-rwx------. 1 root root 1243864 Aug? 1? 2020 shimx64.efi

and we see that the grub.cfg is being updated.

However, here the grubenv file contains

# GRUB Environment Block
saved_entry=CentOS Linux (3.10.0-1160.36.2.el7.x86_64) 7 (Core)
#######################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################

So maybe I have the reason.

Now how to cure??

I can obviously edit this file to point to the latest kernel, but this 
would likely need to be done each time I update the system and get a new 
kernel - not my preferred option.

How come the BIOS boot files are updated each time, but not the UEFI 
based one?? Well grub.cfg is, but not grubenv

Can I edit /etc/default/grub and change

GRUB_DEFAULT=saved

to something else?

Stumped.

I think I have some basic UEFI install stuff missing, but back when I 
manually changed the boot system from BIOS to UEFI I was told only 
anaconda does this at install time, and I was unwilling to do a complete 
reinstall on an already in production system. So after much trial, I did 
get it booting reliably, just this one issues remains.

Any pointers appreciated.

TIA Rob.

Change it to

GRUB_DEFAULT=0

(I encountered the same issue week ago with a workstation booted for three month
with an older
kernel because of https://bugzilla.redhat.com/show_bug.cgi?id=2143438 , and
solved it this way)

Regards,

Petko



On 3/14/23 10:51, Rob Kampen wrote:
> Can I edit /etc/default/grub and change
> 
> GRUB_DEFAULT=saved
> 
> to something else?
-- 
Petko Alov

Department of QSAR & Molecular Modelling
Institute of Biophysics and Biomedical Engineering, BAS
21 G. Bontchev Str
1113 Sofia, BULGARIA

phone:  +359  2 9793647
mobile: +359 87 7294336
e-mail: petko.alov at biophys.bas.bg