CentOS - Aug 2020 - 8.2.2004 Latest yum update renders machine unbootable

On 8/2/20 2:04 AM, Alessandro Baggi wrote:
> 
> Il 01/08/20 22:03, Greg Bailey ha scritto:
>> On 8/1/20 6:56 AM, david wrote:
>>> At 02:54 AM 8/1/2020, Alessandro Baggi wrote:
>>>> Hi Johnny,
>>>> thank you very much for clarification.
>>>>
>>>> You said that in the centos infrastructure only one server got
the
>>>> problem.
>>>> What are the conditions that permit the breakage? There is a
particular
>>>> configuration (hw/sw) case that match always the problem or it
is
>>>> random?
>>>>
>>>> Thank you
>>>
>>> I have two servers running Centos 7 on apple hardware (one mac-mini
>>> and one mac server).? They both failed to reboot a few days ago.?
So
>>> perhaps whatever anti-boot bug hit Centos 8, also hit Centos 7.? I
>>> can't tell what version got updated since the system simply
fails to
>>> boot.? I don't even get a grub screen. I'll have to rebuild
the
>>> systems from scratch.
>>>
>>>
>>
>> You should be able to boot off of installation media into rescue mode,
>> and downgrade the grub2* and/or shim* RPMs.
>>
>> -Greg
>>
> I did the downgrade on a fresh install of c8.2 but yum said that all
> selected packages (grub2,shim...) are already to the lowest version and
> the downgrade is not possibile, ending with "nothing to do".
Ok .. We are running through some final testing now for CentOS Linux 8
and CentOS Stream .. updates later today for EL8.

For CentOS Linux 7 .. I just pushed the latest shim packages (we had to
get these signed by Microsoft .. as do all distros that do shim.
Microsoft is the official CA for secureboot.

So in the next few hours, after the mirrors sync up .. you should be
able to fix any EL7 machines.

I'll post here again once we have pushed the EL8 and CentOS Stream updates.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20200802/36f59c5b/attachment.sig>

At Sun, 2 Aug 2020 06:59:06 -0500 CentOS mailing list <centos at
centos.org> wrote:
> 
> 
> 
> On 8/2/20 2:04 AM, Alessandro Baggi wrote:
> > 
> > Il 01/08/20 22:03, Greg Bailey ha scritto:
> >> On 8/1/20 6:56 AM, david wrote:
> >>> At 02:54 AM 8/1/2020, Alessandro Baggi wrote:
> >>>> Hi Johnny,
> >>>> thank you very much for clarification.
> >>>>
> >>>> You said that in the centos infrastructure only one server
got the
> >>>> problem.
> >>>> What are the conditions that permit the breakage? There is
a particular
> >>>> configuration (hw/sw) case that match always the problem
or it is
> >>>> random?
> >>>>
> >>>> Thank you
> >>>
> >>> I have two servers running Centos 7 on apple hardware (one
mac-mini
> >>> and one mac server).???????? They both failed to reboot a few
days ago.???????? So
> >>> perhaps whatever anti-boot bug hit Centos 8, also hit Centos
7.???????? I
> >>> can't tell what version got updated since the system
simply fails to
> >>> boot.???????? I don't even get a grub screen. I'll
have to rebuild the
> >>> systems from scratch.
> >>>
> >>>
> >>
> >> You should be able to boot off of installation media into rescue
mode,
> >> and downgrade the grub2* and/or shim* RPMs.
> >>
> >> -Greg
> >>
> > I did the downgrade on a fresh install of c8.2 but yum said that all
> > selected packages (grub2,shim...) are already to the lowest version
and
> > the downgrade is not possibile, ending with "nothing to do".
> 
> Ok .. We are running through some final testing now for CentOS Linux 8
> and CentOS Stream .. updates later today for EL8.
> 
> For CentOS Linux 7 .. I just pushed the latest shim packages (we had to
> get these signed by Microsoft .. as do all distros that do shim.
> Microsoft is the official CA for secureboot.
> 
> So in the next few hours, after the mirrors sync up .. you should be
> able to fix any EL7 machines.
Question: is this only a problem for bare metal w/EFI or are VMs affected?  I 
have a VPS running CentOS 7:

sharky4.deepsoft.com% uname -a
Linux sharky4.deepsoft.com 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 
15:46:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
sharky4.deepsoft.com% rpm -qa grub\* shim\*
grub2-tools-extra-2.02-0.81.el7.centos.x86_64
grub2-pc-modules-2.02-0.81.el7.centos.noarch
grub2-tools-minimal-2.02-0.81.el7.centos.x86_64
grub2-2.02-0.81.el7.centos.x86_64
grub2-tools-2.02-0.81.el7.centos.x86_64
grubby-8.28-26.el7.x86_64
grub2-common-2.02-0.81.el7.centos.noarch
grub2-pc-2.02-0.81.el7.centos.x86_64
sharky4.deepsoft.com% 

I have these (pending) updates:

sharky4.deepsoft.com% sudo /usr/bin/yum check-update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.es.its.nyu.edu
  * epel: mirror.math.princeton.edu
   * extras: mirror.facebook.net
    * updates: mirror.atlanticmetro.net
    
fail2ban.noarch                      0.11.1-9.el7.2                    epel   
fail2ban-firewalld.noarch            0.11.1-9.el7.2                    epel   
fail2ban-sendmail.noarch             0.11.1-9.el7.2                    epel   
fail2ban-server.noarch               0.11.1-9.el7.2                    epel   
fail2ban-systemd.noarch              0.11.1-9.el7.2                    epel   
grub2.x86_64                         1:2.02-0.86.el7.centos            updates
grub2-common.noarch                  1:2.02-0.86.el7.centos            updates
grub2-pc.x86_64                      1:2.02-0.86.el7.centos            updates
grub2-pc-modules.noarch              1:2.02-0.86.el7.centos            updates
grub2-tools.x86_64                   1:2.02-0.86.el7.centos            updates
grub2-tools-extra.x86_64             1:2.02-0.86.el7.centos            updates
grub2-tools-minimal.x86_64           1:2.02-0.86.el7.centos            updates
kernel.x86_64                        3.10.0-1127.18.2.el7              updates
kernel-headers.x86_64                3.10.0-1127.18.2.el7              updates
kernel-tools.x86_64                  3.10.0-1127.18.2.el7              updates
kernel-tools-libs.x86_64             3.10.0-1127.18.2.el7              updates
python-perf.x86_64                   3.10.0-1127.18.2.el7              updates

Is it "safe" for me to to do a yum update or should I wait?


> 
> I'll post here again once we have pushed the EL8 and CentOS Stream
updates.
> 
> Content-Description: OpenPGP digital signature
> 
> -----BEGIN PGP SIGNATURE-----
> 
> iF0EARECAB0WIQTn6goIPoKGmzXde4tMqQyCasFjswUCXyaqigAKCRBMqQyCasFj
> swMtAKCIljOaB6o0mxnvIUqA0pP2l16hUwCgnmSj3aPKOym7s58ismQ0mDKfwus> =S/HK
> -----END PGP SIGNATURE-----
> 
> MIME-Version: 1.0
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
> 
>
> 
-- 
Robert Heller             -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services

On 8/2/20 7:06 AM, Robert Heller wrote:
> At Sun, 2 Aug 2020 06:59:06 -0500 CentOS mailing list <centos at
centos.org> wrote:
> 
>>
>>
>>
>> On 8/2/20 2:04 AM, Alessandro Baggi wrote:
>>>
>>> Il 01/08/20 22:03, Greg Bailey ha scritto:
>>>> On 8/1/20 6:56 AM, david wrote:
>>>>> At 02:54 AM 8/1/2020, Alessandro Baggi wrote:
>>>>>> Hi Johnny,
>>>>>> thank you very much for clarification.
>>>>>>
>>>>>> You said that in the centos infrastructure only one
server got the
>>>>>> problem.
>>>>>> What are the conditions that permit the breakage? There
is a particular
>>>>>> configuration (hw/sw) case that match always the
problem or it is
>>>>>> random?
>>>>>>
>>>>>> Thank you
>>>>>
>>>>> I have two servers running Centos 7 on apple hardware (one
mac-mini
>>>>> and one mac server).???????? They both failed to reboot a
few days ago.???????? So
>>>>> perhaps whatever anti-boot bug hit Centos 8, also hit
Centos 7.???????? I
>>>>> can't tell what version got updated since the system
simply fails to
>>>>> boot.???????? I don't even get a grub screen. I'll
have to rebuild the
>>>>> systems from scratch.
>>>>>
>>>>>
>>>>
>>>> You should be able to boot off of installation media into
rescue mode,
>>>> and downgrade the grub2* and/or shim* RPMs.
>>>>
>>>> -Greg
>>>>
>>> I did the downgrade on a fresh install of c8.2 but yum said that
all
>>> selected packages (grub2,shim...) are already to the lowest version
and
>>> the downgrade is not possibile, ending with "nothing to
do".
>>
>> Ok .. We are running through some final testing now for CentOS Linux 8
>> and CentOS Stream .. updates later today for EL8.
>>
>> For CentOS Linux 7 .. I just pushed the latest shim packages (we had to
>> get these signed by Microsoft .. as do all distros that do shim.
>> Microsoft is the official CA for secureboot.
>>
>> So in the next few hours, after the mirrors sync up .. you should be
>> able to fix any EL7 machines.
> 
> Question: is this only a problem for bare metal w/EFI or are VMs affected? 
I
> have a VPS running CentOS 7:
> 
> sharky4.deepsoft.com% uname -a
> Linux sharky4.deepsoft.com 3.10.0-1127.13.1.el7.x86_64 #1 SMP Tue Jun 23 
> 15:46:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> sharky4.deepsoft.com% rpm -qa grub\* shim\*
> grub2-tools-extra-2.02-0.81.el7.centos.x86_64
> grub2-pc-modules-2.02-0.81.el7.centos.noarch
> grub2-tools-minimal-2.02-0.81.el7.centos.x86_64
> grub2-2.02-0.81.el7.centos.x86_64
> grub2-tools-2.02-0.81.el7.centos.x86_64
> grubby-8.28-26.el7.x86_64
> grub2-common-2.02-0.81.el7.centos.noarch
> grub2-pc-2.02-0.81.el7.centos.x86_64
> sharky4.deepsoft.com% 
> 
> I have these (pending) updates:
> 
> sharky4.deepsoft.com% sudo /usr/bin/yum check-update
> Loaded plugins: fastestmirror
> Loading mirror speeds from cached hostfile
>  * base: mirror.es.its.nyu.edu
>   * epel: mirror.math.princeton.edu
>    * extras: mirror.facebook.net
>     * updates: mirror.atlanticmetro.net
>     
> fail2ban.noarch                      0.11.1-9.el7.2                    epel
> fail2ban-firewalld.noarch            0.11.1-9.el7.2                    epel
> fail2ban-sendmail.noarch             0.11.1-9.el7.2                    epel
> fail2ban-server.noarch               0.11.1-9.el7.2                    epel
> fail2ban-systemd.noarch              0.11.1-9.el7.2                    epel
> grub2.x86_64                         1:2.02-0.86.el7.centos           
updates
> grub2-common.noarch                  1:2.02-0.86.el7.centos           
updates
> grub2-pc.x86_64                      1:2.02-0.86.el7.centos           
updates
> grub2-pc-modules.noarch              1:2.02-0.86.el7.centos           
updates
> grub2-tools.x86_64                   1:2.02-0.86.el7.centos           
updates
> grub2-tools-extra.x86_64             1:2.02-0.86.el7.centos           
updates
> grub2-tools-minimal.x86_64           1:2.02-0.86.el7.centos           
updates
> kernel.x86_64                        3.10.0-1127.18.2.el7             
updates
> kernel-headers.x86_64                3.10.0-1127.18.2.el7             
updates
> kernel-tools.x86_64                  3.10.0-1127.18.2.el7             
updates
> kernel-tools-libs.x86_64             3.10.0-1127.18.2.el7             
updates
> python-perf.x86_64                   3.10.0-1127.18.2.el7             
updates
> 
> Is it "safe" for me to to do a yum update or should I wait?
> 
> 
> 
This shim issue should only impact cold iron machines with secureboot
enabled.




-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20200802/b2083d75/attachment.sig>

On 8/2/20 6:59 AM, Johnny Hughes wrote:
> On 8/2/20 2:04 AM, Alessandro Baggi wrote:
>>
>> Il 01/08/20 22:03, Greg Bailey ha scritto:
>>> On 8/1/20 6:56 AM, david wrote:
>>>> At 02:54 AM 8/1/2020, Alessandro Baggi wrote:
>>>>> Hi Johnny,
>>>>> thank you very much for clarification.
>>>>>
>>>>> You said that in the centos infrastructure only one server
got the
>>>>> problem.
>>>>> What are the conditions that permit the breakage? There is
a particular
>>>>> configuration (hw/sw) case that match always the problem or
it is
>>>>> random?
>>>>>
>>>>> Thank you
>>>>
>>>> I have two servers running Centos 7 on apple hardware (one
mac-mini
>>>> and one mac server).? They both failed to reboot a few days
ago.? So
>>>> perhaps whatever anti-boot bug hit Centos 8, also hit Centos
7.? I
>>>> can't tell what version got updated since the system simply
fails to
>>>> boot.? I don't even get a grub screen. I'll have to
rebuild the
>>>> systems from scratch.
>>>>
>>>>
>>>
>>> You should be able to boot off of installation media into rescue
mode,
>>> and downgrade the grub2* and/or shim* RPMs.
>>>
>>> -Greg
>>>
>> I did the downgrade on a fresh install of c8.2 but yum said that all
>> selected packages (grub2,shim...) are already to the lowest version and
>> the downgrade is not possibile, ending with "nothing to do".
> 
> Ok .. We are running through some final testing now for CentOS Linux 8
> and CentOS Stream .. updates later today for EL8.
> 
> For CentOS Linux 7 .. I just pushed the latest shim packages (we had to
> get these signed by Microsoft .. as do all distros that do shim.
> Microsoft is the official CA for secureboot.
> 
> So in the next few hours, after the mirrors sync up .. you should be
> able to fix any EL7 machines.
> 
> I'll post here again once we have pushed the EL8 and CentOS Stream
updates.
OK .. I have also now pushed the CentOS Linux 8 update .. you should see
an update to SHIM .. the new versions are:

PowerTools/x86_64/os/Packages/shim-unsigned-x64-15-8.el8.x86_64.rpm
BaseOS/x86_64/os/Packages/shim-ia32-15-15.el8_2.x86_64.rpm
BaseOS/x86_64/os/Packages/shim-x64-15-15.el8_2.x86_64.rpm

For CentOS Linux 7 .. the new files are:

x86_64/Packages/mokutil-15-8.el7.x86_64.rpm
x86_64/Packages/shim-ia32-15-8.el7.x86_64.rpm
x86_64/Packages/shim-unsigned-ia32-15-8.el7.x86_64.rpm
x86_64/Packages/shim-unsigned-x64-15-8.el7.x86_64.rpm
x86_64/Packages/shim-x64-15-8.el7.x86_64.rpm

You need only replace the files you currently have installed, not
install every file.

Please report both positive and negative results.

Thanks,
Johnny Hughes


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20200802/fab5a7f2/attachment.sig>

> Am 02.08.2020 um 14:34 schrieb Johnny Hughes <johnny at centos.org>:
> 
> ?On 8/2/20 6:59 AM, Johnny Hughes wrote:
>>> On 8/2/20 2:04 AM, Alessandro Baggi wrote:
>>> 
>>> Il 01/08/20 22:03, Greg Bailey ha scritto:
>>>> On 8/1/20 6:56 AM, david wrote:
>>>>> At 02:54 AM 8/1/2020, Alessandro Baggi wrote:
>>>>>> Hi Johnny,
>>>>>> thank you very much for clarification.
>>>>>> 
>>>>>> You said that in the centos infrastructure only one
server got the
>>>>>> problem.
>>>>>> What are the conditions that permit the breakage? There
is a particular
>>>>>> configuration (hw/sw) case that match always the
problem or it is
>>>>>> random?
>>>>>> 
>>>>>> Thank you
>>>>> 
>>>>> I have two servers running Centos 7 on apple hardware (one
mac-mini
>>>>> and one mac server).  They both failed to reboot a few days
ago.  So
>>>>> perhaps whatever anti-boot bug hit Centos 8, also hit
Centos 7.  I
>>>>> can't tell what version got updated since the system
simply fails to
>>>>> boot.  I don't even get a grub screen. I'll have to
rebuild the
>>>>> systems from scratch.
>>>>> 
>>>>> 
>>>> 
>>>> You should be able to boot off of installation media into
rescue mode,
>>>> and downgrade the grub2* and/or shim* RPMs.
>>>> 
>>>> -Greg
>>>> 
>>> I did the downgrade on a fresh install of c8.2 but yum said that
all
>>> selected packages (grub2,shim...) are already to the lowest version
and
>>> the downgrade is not possibile, ending with "nothing to
do".
>> 
>> Ok .. We are running through some final testing now for CentOS Linux 8
>> and CentOS Stream .. updates later today for EL8.
>> 
>> For CentOS Linux 7 .. I just pushed the latest shim packages (we had to
>> get these signed by Microsoft .. as do all distros that do shim.
>> Microsoft is the official CA for secureboot.
>> 
>> So in the next few hours, after the mirrors sync up .. you should be
>> able to fix any EL7 machines.
>> 
>> I'll post here again once we have pushed the EL8 and CentOS Stream
updates.
> 
> OK .. I have also now pushed the CentOS Linux 8 update .. you should see
> an update to SHIM .. the new versions are:
> 
> PowerTools/x86_64/os/Packages/shim-unsigned-x64-15-8.el8.x86_64.rpm
> BaseOS/x86_64/os/Packages/shim-ia32-15-15.el8_2.x86_64.rpm
> BaseOS/x86_64/os/Packages/shim-x64-15-15.el8_2.x86_64.rpm
> 
> For CentOS Linux 7 .. the new files are:
> 
> x86_64/Packages/mokutil-15-8.el7.x86_64.rpm
> x86_64/Packages/shim-ia32-15-8.el7.x86_64.rpm
> x86_64/Packages/shim-unsigned-ia32-15-8.el7.x86_64.rpm
> x86_64/Packages/shim-unsigned-x64-15-8.el7.x86_64.rpm
> x86_64/Packages/shim-x64-15-8.el7.x86_64.rpm
> 
> You need only replace the files you currently have installed, not
> install every file.
> 
> Please report both positive and negative results.
A previously afftected Aures nino POS-terminal boots just fine with CentOS 7 and
shim 15.8.

Now I will test an affected machine (Aures Twist POS terminal) that runs CentOS
8 (well, usually runs CentOS 8, but not now, since it does not boot ... :=>
> 
> Thanks,
> Johnny Hughes
> 
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos

><snip>

> > I'll post here again once we have pushed the EL8 and CentOS Stream
updates.
>
>OK .. I have also now pushed the CentOS Linux 8 update .. you should see
>an update to SHIM .. the new versions are:
>
>PowerTools/x86_64/os/Packages/shim-unsigned-x64-15-8.el8.x86_64.rpm
>BaseOS/x86_64/os/Packages/shim-ia32-15-15.el8_2.x86_64.rpm
>BaseOS/x86_64/os/Packages/shim-x64-15-15.el8_2.x86_64.rpm
>
>For CentOS Linux 7 .. the new files are:
>
>x86_64/Packages/mokutil-15-8.el7.x86_64.rpm
>x86_64/Packages/shim-ia32-15-8.el7.x86_64.rpm
>x86_64/Packages/shim-unsigned-ia32-15-8.el7.x86_64.rpm
>x86_64/Packages/shim-unsigned-x64-15-8.el7.x86_64.rpm
>x86_64/Packages/shim-x64-15-8.el7.x86_64.rpm
>
>You need only replace the files you currently have installed, not
>install every file.
>
>Please report both positive and negative results.
>
>Thanks,
>Johnny Hughes
>
I updated my Centos 7, and it's awaiting a reboot.  The version of 
shim says 15-7, which I presume is the bad one.

What do I need to do?  Obviously, a reboot will encounter the 
failure.  Or, should I perform "yum update" periodically and wait for 
shim 15-8 to appear, and only after that do the reboot?  Or is there 
some other remedy?

And by the way, I'm amazed at the speed with which this problem was 
addressed and a fix provided.  You guys work wonders.

David