CentOS - May 2018 - CentOS6: HELP! EFI boot fails after replacing disks...

OK, I wanted to replace the 500G disks in a Dell T20  server with new 2TB 
disks.  The machine has 4 SATA ports, one used for the optical disk and three 
for the hard drives.  It is set up with /dev/sda and /dev/sdb with each three 
partitions:

1 -- VFAT (for EFI)
2 -- ext4 (for /boot)
3 -- LVM

/dev/sda2 and /dev/sdb2 are a mirror raid (/dev/md0)
/dev/sda3 and /dev/sdb3 are a mirror raid (/dev/md1), with LVM on top of that.

/dev/sdc is a disk containing one file system and mostly used by AMANDA for 
backup (it has a "virtual" tape changer).

This morning I shut the machine down and pulled [the 500G] /dev/sdb and
installed a new 2TB disk as a new /dev/sdb. Partitioned it (with parted) to be
much like /dev/sda (except partition 3 is way bigger). I added the new
/dev/sdb2 to /dev/sda2 (boot raid set) and dd'ed /dev/sda1 to /dev/sdb1. I
then created a new RAID set (degraded) with /dev/sdb3, used pvcreate on it,
used vgextend to add it to the system volume group, then used pvmove to move
the extents from the old disk to the new disk. Meanwhile I partitioned a
second 2TB disk using a USB SATA dock and copied the old 500G /dev/sdc1 to the
new 2TB disk. So far so good.

Then I shut the machine down, swapped in the new backup [2TB] disk and pulled
the old system [500G] disk and installed the third new [2TB] disk. The system
won't boot that way. It seems there is something in the UEFI (secure boot)
logic that wants the original disk, even if everything is moved over.

I ended up putting the original /dev/sda in.  The machine boots and is using 
the new system disk (a raid array in degraded mode).

What is the magic to fix this?  

I tried to run efibootmgr, but it wants a model named efivars loaded, but 
there is no such module available.

What am I missing?

-- 
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services

OK, one other tidbit:

The EFI BIOS has a UUID in its boot options.  I expect that this identifies 
the old system disk, but I don't know where that UUID comes from.  It is NOT
the VFAT UUID for the EFI partition and is not any of the UUIDs for any of the 
Linux file systems or RAID arrays, or really anything else I can find under 
Linux.  I'm guessing it is something the EFI BIOS has come up with, but I am
not sure what exactly.  *I* don't remember filling that in -- I think
anaconda
filled it in during the install process, so presumably, there is some magic 
under Linux to get this UUID (for the new disk(s)) and fill it in, but I 
cannot figure out what.  Maybe efibootmgr has something to do with it, but 
efibootmgr does not work, either on a live system or a system booted from a 
DVD (CentOS 6.9 boot/install DVD) -- efibootmgr wants a kernel module 
named efivars that does not seem to exist.

At Mon, 28 May 2018 18:25:39 -0400 (EDT) CentOS mailing list <centos at
centos.org> wrote:
> 
> OK, I wanted to replace the 500G disks in a Dell T20  server with new 2TB 
> disks.  The machine has 4 SATA ports, one used for the optical disk and
three
> for the hard drives.  It is set up with /dev/sda and /dev/sdb with each
three
> partitions:
> 
> 1 -- VFAT (for EFI)
> 2 -- ext4 (for /boot)
> 3 -- LVM
> 
> /dev/sda2 and /dev/sdb2 are a mirror raid (/dev/md0)
> /dev/sda3 and /dev/sdb3 are a mirror raid (/dev/md1), with LVM on top of
that.
> 
> /dev/sdc is a disk containing one file system and mostly used by AMANDA for
> backup (it has a "virtual" tape changer).
> 
> This morning I shut the machine down and pulled [the 500G] /dev/sdb and
> installed a new 2TB disk as a new /dev/sdb. Partitioned it (with parted) to
be
> much like /dev/sda (except partition 3 is way bigger). I added the new
> /dev/sdb2 to /dev/sda2 (boot raid set) and dd'ed /dev/sda1 to
/dev/sdb1. I
> then created a new RAID set (degraded) with /dev/sdb3, used pvcreate on it,
> used vgextend to add it to the system volume group, then used pvmove to
move
> the extents from the old disk to the new disk. Meanwhile I partitioned a
> second 2TB disk using a USB SATA dock and copied the old 500G /dev/sdc1 to
the
> new 2TB disk. So far so good.
> 
> Then I shut the machine down, swapped in the new backup [2TB] disk and
pulled
> the old system [500G] disk and installed the third new [2TB] disk. The
system
> won't boot that way. It seems there is something in the UEFI (secure
boot)
> logic that wants the original disk, even if everything is moved over.
> 
> I ended up putting the original /dev/sda in.  The machine boots and is
using
> the new system disk (a raid array in degraded mode).
> 
> What is the magic to fix this?  
> 
> I tried to run efibootmgr, but it wants a model named efivars loaded, but 
> there is no such module available.
> 
> What am I missing?
> 
-- 
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services

No fix here, but I've had this issue for a while, only "fix" is
not booting with UEFI apparently.

________________________________
From: CentOS <centos-bounces at centos.org> on behalf of Robert Heller
<heller at deepsoft.com>
Sent: Monday, May 28, 2018 6:43:22 PM
To: CentOS mailing list
Subject: Re: [CentOS] CentOS6: HELP! EFI boot fails after replacing disks...

OK, one other tidbit:

The EFI BIOS has a UUID in its boot options.  I expect that this identifies
the old system disk, but I don't know where that UUID comes from.  It is NOT
the VFAT UUID for the EFI partition and is not any of the UUIDs for any of the
Linux file systems or RAID arrays, or really anything else I can find under
Linux.  I'm guessing it is something the EFI BIOS has come up with, but I am
not sure what exactly.  *I* don't remember filling that in -- I think
anaconda
filled it in during the install process, so presumably, there is some magic
under Linux to get this UUID (for the new disk(s)) and fill it in, but I
cannot figure out what.  Maybe efibootmgr has something to do with it, but
efibootmgr does not work, either on a live system or a system booted from a
DVD (CentOS 6.9 boot/install DVD) -- efibootmgr wants a kernel module
named efivars that does not seem to exist.

At Mon, 28 May 2018 18:25:39 -0400 (EDT) CentOS mailing list <centos at
centos.org> wrote:
>
> OK, I wanted to replace the 500G disks in a Dell T20  server with new 2TB
> disks.  The machine has 4 SATA ports, one used for the optical disk and
three
> for the hard drives.  It is set up with /dev/sda and /dev/sdb with each
three
> partitions:
>
> 1 -- VFAT (for EFI)
> 2 -- ext4 (for /boot)
> 3 -- LVM
>
> /dev/sda2 and /dev/sdb2 are a mirror raid (/dev/md0)
> /dev/sda3 and /dev/sdb3 are a mirror raid (/dev/md1), with LVM on top of
that.
>
> /dev/sdc is a disk containing one file system and mostly used by AMANDA for
> backup (it has a "virtual" tape changer).
>
> This morning I shut the machine down and pulled [the 500G] /dev/sdb and
> installed a new 2TB disk as a new /dev/sdb. Partitioned it (with parted) to
be
> much like /dev/sda (except partition 3 is way bigger). I added the new
> /dev/sdb2 to /dev/sda2 (boot raid set) and dd'ed /dev/sda1 to
/dev/sdb1. I
> then created a new RAID set (degraded) with /dev/sdb3, used pvcreate on it,
> used vgextend to add it to the system volume group, then used pvmove to
move
> the extents from the old disk to the new disk. Meanwhile I partitioned a
> second 2TB disk using a USB SATA dock and copied the old 500G /dev/sdc1 to
the
> new 2TB disk. So far so good.
>
> Then I shut the machine down, swapped in the new backup [2TB] disk and
pulled
> the old system [500G] disk and installed the third new [2TB] disk. The
system
> won't boot that way. It seems there is something in the UEFI (secure
boot)
> logic that wants the original disk, even if everything is moved over.
>
> I ended up putting the original /dev/sda in.  The machine boots and is
using
> the new system disk (a raid array in degraded mode).
>
> What is the magic to fix this?
>
> I tried to run efibootmgr, but it wants a model named efivars loaded, but
> there is no such module available.
>
> What am I missing?
>
--
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services

_______________________________________________
CentOS mailing list
CentOS at centos.org
https://lists.centos.org/mailman/listinfo/centos

On May 28, 2018, at 18:25, Robert Heller <heller at deepsoft.com> wrote:
> I tried to run efibootmgr, but it wants a model named efivars loaded, but 
> there is no such module available.
Are you not running a CentOS kernel?  That module should be available. 

The UUID of the VFAT volume (not the mirror) would be used in the EFI boot
entry. Boot off a rescue disk when the new disk is installed and add an
additional boot entry for the new disk. It will reflect the UUID of the EFI
partition on the new disk. Run ?blkid? to compare.

--
Jonathan Billings

At Mon, 28 May 2018 19:30:25 -0400 CentOS mailing list <centos at
centos.org> wrote:
> 
> On May 28, 2018, at 18:25, Robert Heller <heller at deepsoft.com>
wrote:
> 
> > I tried to run efibootmgr, but it wants a model named efivars loaded,
but
> > there is no such module available.
> 
> Are you not running a CentOS kernel?  That module should be available. 
I have a *stock* CentOS 6 kernel:  2.6.32-696.28.1.el6.x86_64.  And no, that 
module is not in a CentOS Plus kernel either.
> 
> The UUID of the VFAT volume (not the mirror) would be used in the EFI boot
entry. Boot off a rescue disk when the new disk is installed and add an
additional boot entry for the new disk. It will reflect the UUID of the EFI
partition on the new disk. Run ???blkid??? to compare.
> 
The UUID in the BIOS is NOT VFAT volume.  It is something completely 
different.  I have no clue what it is -- it does not correspond to anything I 
can find.

> --
> Jonathan Billings
> 
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
> 
>                               
-- 
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services

On 05/28/2018 03:25 PM, Robert Heller wrote:
> I tried to run efibootmgr, but it wants a model named efivars loaded, but
> there is no such module available.
That's interesting.? Can you post the command and output where you see that?

Also, post the output of "dmesg | grep efi:" and "ls
/sys/firmware/efi/"

At Mon, 28 May 2018 18:23:42 -0700 CentOS mailing list <centos at
centos.org> wrote:
> 
> On 05/28/2018 03:25 PM, Robert Heller wrote:
> > I tried to run efibootmgr, but it wants a model named efivars loaded,
but
> > there is no such module available.
> 
> That's interesting.?? Can you post the command and output where you see
that?
> 
> Also, post the output of "dmesg | grep efi:" and "ls
/sys/firmware/efi/"
newserver.wendellfreelibrary.org% sudo efibootmgr
[sudo] password for heller: 
Fatal: Couldn't open either sysfs or procfs directories for accessing EFI
variables.
Try 'modprobe efivars' as root.
newserver.wendellfreelibrary.org% sudo  modprobe efivars
FATAL: Module efivars not found.
newserver.wendellfreelibrary.org% dmesg | grep efi:
newserver.wendellfreelibrary.org% ls /sys/firmware/efi/
ls: cannot access /sys/firmware/efi/: No such file or directory
newserver.wendellfreelibrary.org% uname -a
Linux newserver.wendellfreelibrary.org 2.6.32-696.28.1.el6.x86_64 #1 SMP Wed 
May 9 23:09:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
newserver.wendellfreelibrary.org% 



> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
> 
>
-- 
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services

At Mon, 28 May 2018 18:23:42 -0700 CentOS mailing list <centos at
centos.org> wrote:
> 
> On 05/28/2018 03:25 PM, Robert Heller wrote:
> > I tried to run efibootmgr, but it wants a model named efivars loaded,
but
> > there is no such module available.
> 
> That's interesting.?? Can you post the command and output where you see
that?
> 
> Also, post the output of "dmesg | grep efi:" and "ls
/sys/firmware/efi/"
And for completeness:

newserver.wendellfreelibrary.org% rpm -qi kernel-2.6.32-696.28.1.el6 
Name        : kernel                       Relocations: (not relocatable)
Version     : 2.6.32                            Vendor: CentOS
Release     : 696.28.1.el6                  Build Date: Wed 09 May 2018 07:26:32
PM EDT
Install Date: Fri 18 May 2018 03:58:09 PM EDT      Build Host:
x86-01.bsys.centos.org
Group       : System Environment/Kernel     Source RPM:
kernel-2.6.32-696.28.1.el6.src.rpm
Size        : 139572226                        License: GPLv2
Signature   : RSA/SHA1, Wed 09 May 2018 08:59:22 PM EDT, Key ID 0946fca2c105b9de
Packager    : CentOS BuildSystem <http://bugs.centos.org>
URL         : http://www.kernel.org/
Summary     : The Linux kernel
Description :
The kernel package contains the Linux kernel (vmlinuz), the core of any
Linux operating system.  The kernel handles the basic functions
of the operating system: memory allocation, process allocation, device
input and output, etc.

> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
> 
>
> 
-- 
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services

On 28 May 2018 at 18:25, Robert Heller <heller at deepsoft.com> wrote:
> Then I shut the machine down, swapped in the new backup [2TB] disk and
pulled
> the old system [500G] disk and installed the third new [2TB] disk. The
system
> won't boot that way. It seems there is something in the UEFI (secure
boot)
> logic that wants the original disk, even if everything is moved over.
>
This caught my attention. Did you mean that you are using the secure
boot options? I don't know if that ties down a system to a specific
disk until it is cleared from the install. From all the other items
you listed in the the thread, your system looks like it is booting
into a form where it is saying it isn't UEFI anymore which would be a
boot firmware option. The firmware can lock down what it thinks is ok
to boot from and may require some sort of flush depending on the type
of disks it thinks are ok. I had this with one set of systems where
the system required a hard flush of UEFI buffers before it would boot
from a larger disk. It was ok with the same old model but a new one
was not possible.

My debugging methodology at this point would be the following:

1. Boot from EL6 iso and see if EFI variables/modules work
2. Boot from EL7 iso and see if EFI variables/modules work
3. See if a Dell firmware update set is available and if the
changelogs say it fixes EFI boot issues
4. repeat 1&2 if a firmware update is done.
>From what I can tell, most of the install methods and 0 downtime
'hacks' we have used for the last 30 years on BIOS systems are either
impossible or need serious fixes to work again in a UEFI world.


-- 
Stephen J Smoogen.

At Tue, 29 May 2018 10:03:07 -0400 CentOS mailing list <centos at
centos.org> wrote:
> 
> On 28 May 2018 at 18:25, Robert Heller <heller at deepsoft.com>
wrote:
> 
> > Then I shut the machine down, swapped in the new backup [2TB] disk and
pulled
> > the old system [500G] disk and installed the third new [2TB] disk. The
system
> > won't boot that way. It seems there is something in the UEFI
(secure boot)
> > logic that wants the original disk, even if everything is moved over.
> >
> 
> This caught my attention. Did you mean that you are using the secure
> boot options? I don't know if that ties down a system to a specific
> disk until it is cleared from the install. From all the other items
> you listed in the the thread, your system looks like it is booting
> into a form where it is saying it isn't UEFI anymore which would be a
> boot firmware option. The firmware can lock down what it thinks is ok
> to boot from and may require some sort of flush depending on the type
> of disks it thinks are ok. I had this with one set of systems where
> the system required a hard flush of UEFI buffers before it would boot
> from a larger disk. It was ok with the same old model but a new one
> was not possible.
Not using "secure boot" in the sense of setting any partitular
security (the
"secure" section of the  BIOS is not enabled.  What *was* enabled was
the UEFI
boot.  At this point I have mostly given up on UEFI.  I disconnected the 
optical disk (we don't really use it anyway) and connected the original boot
disk as /dev/sdd and installed the third 2TB disk.  The system boots (in 
Legacy Mode) and it is now rebuilding the RAID array onto the new disk.  The 
/boot array now has three elements (partition 2 of the two new disks and 
partition 2 of the old disk).  I condemed the old (empty) RAID array on the 
old disk -- it now has an unformatted third partition that is not being used 
for anything.  I will be *eventually* upgrading the system to CentOS 7 
(sometime later this summer).  Maybe at that time I will revisit the world of 
UEFI...
> 
> My debugging methodology at this point would be the following:
> 
> 1. Boot from EL6 iso and see if EFI variables/modules work
> 2. Boot from EL7 iso and see if EFI variables/modules work
> 3. See if a Dell firmware update set is available and if the
> changelogs say it fixes EFI boot issues
> 4. repeat 1&2 if a firmware update is done.
> 
> >From what I can tell, most of the install methods and 0 downtime
> 'hacks' we have used for the last 30 years on BIOS systems are
either
> impossible or need serious fixes to work again in a UEFI world.
> 
> 
-- 
Robert Heller             -- 978-544-6933
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services