CentOS - Aug 2020 - 8.2.2004 Latest yum update renders machine unbootable

Il 31/07/20 13:08, ja ha scritto:
> On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
>> I am running an Intel x64 machine using UEFI to boot an SSD.
>>
>> Installing the latest yum update which includes grub2 and kernel
>> 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank
>> screen where grub should be, no error messages, just hangs.
>>
>> After some hours I managed to modify another bootable partition
>> (containing older software) and boot it from there.
>>
>> After that, I  found out it is a known problem.
>>
>> The main point of this message is to make people aware of the problem
>> and suggest admins don't run 'yum update' until they
understand the
>> problem and have a fix at hand.
>>
>> See 'UEFI boot blank screen post update' for a solution and
directions
>> to the redhat article.
>>
>> Regards
>>
>> Alan
>>
> I have been punished by this bug - it is/was very nasty.
>
Me too. Luckily it happened on a test machine.

Sorry but seems that those packages were not tested before pushing them 
in the update repo. Would be great to know what happened to the 
mainstream chains and how a package like grub reached the update repo 
when it has serious problem (genuine curiosity but not to blame them).

In my case, the restore procedure reported by RH in case of reboot does 
not work and reports that the packages are already at the lowest version 
and that the downgrade is not possibile. I don't know why.

RedHat is recommending not to patch.
https://access.redhat.com/solutions/5272311

Andrea

-----Original Message-----
From: CentOS <centos-bounces at centos.org> On Behalf Of Alessandro Baggi
Sent: Friday, July 31, 2020 11:25 AM
To: centos at centos.org
Subject: {EXTERNAL} Re: [CentOS] 8.2.2004 Latest yum update renders machine
unbootable


CAUTION:??This email originated outside of BSWH; avoid action unless you know
the content is safe. Send suspicious emails as attachments to BadEmail at
BSWHealth.org.


Il 31/07/20 13:08, ja ha scritto:
> On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
>> I am running an Intel x64 machine using UEFI to boot an SSD.
>>
>> Installing the latest yum update which includes grub2 and kernel
>> 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank 
>> screen where grub should be, no error messages, just hangs.
>>
>> After some hours I managed to modify another bootable partition 
>> (containing older software) and boot it from there.
>>
>> After that, I  found out it is a known problem.
>>
>> The main point of this message is to make people aware of the problem 
>> and suggest admins don't run 'yum update' until they
understand the
>> problem and have a fix at hand.
>>
>> See 'UEFI boot blank screen post update' for a solution and 
>> directions to the redhat article.
>>
>> Regards
>>
>> Alan
>>
> I have been punished by this bug - it is/was very nasty.
>
Me too. Luckily it happened on a test machine.

Sorry but seems that those packages were not tested before pushing them in the
update repo. Would be great to know what happened to the mainstream chains and
how a package like grub reached the update repo when it has serious problem
(genuine curiosity but not to blame them).

In my case, the restore procedure reported by RH in case of reboot does not work
and reports that the packages are already at the lowest version and that the
downgrade is not possibile. I don't know why.

_______________________________________________
CentOS mailing list
CentOS at centos.org
https://urldefense.com/v3/__https://lists.centos.org/mailman/listinfo/centos__;!!JA_k2roV-A!TZ0vdQBvu-4qe6spakWlZ8WocsAa0gEznp4zChtxB-6LsI9T5wEvaXV3KZzeSmSp4Q$

**********************************************************************
The information contained in this e-mail may be privileged and/or confidential,
and protected from disclosure, and no waiver of any attorney-client, work
product, or other privilege is intended.  If you are the intended recipient,
further disclosures are prohibited without proper authorization. If you are not
the intended recipient (or have received this e-mail in error) please notify the
sender immediately and destroy this e-mail. Any unauthorized copying, disclosure
or distribution of the material in this e-mail is strictly forbidden and
possibly a violation of federal or state law and regulations. The sender and
Baylor Scott & White Health, and its affiliated entities, hereby expressly
reserve all privileges and confidentiality that might otherwise be waived as a
result of an erroneous or misdirected e-mail transmission. No employee or agent
is authorized to conclude any binding agreement on behalf of Baylor Scott &
White Health, or any affiliated entity, by e-mail without express written
confirmation by the CEO, the Senior Vice President of Supply Chain Services or
other duly authorized representative of Baylor Scott & White Health.

I would add,

RedHat is recomending also to blacklist grub, shim and mokutil in 
yum.conf until the problem is resolved.

Il 31/07/20 18:26, Laack, Andrea P ha scritto:
> RedHat is recommending not to patch.
> https://access.redhat.com/solutions/5272311
>
> Andrea
>
> -----Original Message-----
> From: CentOS <centos-bounces at centos.org> On Behalf Of Alessandro
Baggi
> Sent: Friday, July 31, 2020 11:25 AM
> To: centos at centos.org
> Subject: {EXTERNAL} Re: [CentOS] 8.2.2004 Latest yum update renders machine
unbootable
>
>
> CAUTION:??This email originated outside of BSWH; avoid action unless you
know the content is safe. Send suspicious emails as attachments to BadEmail at
BSWHealth.org.
>
>
> Il 31/07/20 13:08, ja ha scritto:
>> On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
>>> I am running an Intel x64 machine using UEFI to boot an SSD.
>>>
>>> Installing the latest yum update which includes grub2 and kernel
>>> 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank
>>> screen where grub should be, no error messages, just hangs.
>>>
>>> After some hours I managed to modify another bootable partition
>>> (containing older software) and boot it from there.
>>>
>>> After that, I  found out it is a known problem.
>>>
>>> The main point of this message is to make people aware of the
problem
>>> and suggest admins don't run 'yum update' until they
understand the
>>> problem and have a fix at hand.
>>>
>>> See 'UEFI boot blank screen post update' for a solution and
>>> directions to the redhat article.
>>>
>>> Regards
>>>
>>> Alan
>>>
>> I have been punished by this bug - it is/was very nasty.
>>
> Me too. Luckily it happened on a test machine.
>
> Sorry but seems that those packages were not tested before pushing them in
the update repo. Would be great to know what happened to the mainstream chains
and how a package like grub reached the update repo when it has serious problem
(genuine curiosity but not to blame them).
>
> In my case, the restore procedure reported by RH in case of reboot does not
work and reports that the packages are already at the lowest version and that
the downgrade is not possibile. I don't know why.
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
>
https://urldefense.com/v3/__https://lists.centos.org/mailman/listinfo/centos__;!!JA_k2roV-A!TZ0vdQBvu-4qe6spakWlZ8WocsAa0gEznp4zChtxB-6LsI9T5wEvaXV3KZzeSmSp4Q$
>
> **********************************************************************
> The information contained in this e-mail may be privileged and/or
confidential, and protected from disclosure, and no waiver of any
attorney-client, work product, or other privilege is intended.  If you are the
intended recipient, further disclosures are prohibited without proper
authorization. If you are not the intended recipient (or have received this
e-mail in error) please notify the sender immediately and destroy this e-mail.
Any unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden and possibly a violation of federal or state law
and regulations. The sender and Baylor Scott & White Health, and its
affiliated entities, hereby expressly reserve all privileges and confidentiality
that might otherwise be waived as a result of an erroneous or misdirected e-mail
transmission. No employee or agent is authorized to conclude any binding
agreement on behalf of Baylor Scott & White Health, or any affiliated
entity, by e-mail without express written confirmation by the CEO, the Senior
Vice President of Supply Chain Services or other duly authorized representative
of Baylor Scott & White Health.
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos

On 7/31/20 11:24 AM, Alessandro Baggi wrote:
> 
> Il 31/07/20 13:08, ja ha scritto:
>> On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
>>> I am running an Intel x64 machine using UEFI to boot an SSD.
>>>
>>> Installing the latest yum update which includes grub2 and kernel
>>> 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank
>>> screen where grub should be, no error messages, just hangs.
>>>
>>> After some hours I managed to modify another bootable partition
>>> (containing older software) and boot it from there.
>>>
>>> After that, I? found out it is a known problem.
>>>
>>> The main point of this message is to make people aware of the
problem
>>> and suggest admins don't run 'yum update' until they
understand the
>>> problem and have a fix at hand.
>>>
>>> See 'UEFI boot blank screen post update' for a solution and
directions
>>> to the redhat article.
>>>
>>> Regards
>>>
>>> Alan
>>>
>> I have been punished by this bug - it is/was very nasty.
>>
> Me too. Luckily it happened on a test machine.
> 
> Sorry but seems that those packages were not tested before pushing them
> in the update repo. Would be great to know what happened to the
> mainstream chains and how a package like grub reached the update repo
> when it has serious problem (genuine curiosity but not to blame them).
Of course it was tested before it was pushed. Obviously this is not a
problem with every install.  Surely you don't think we push items
without doing any testing.  Certainly not items as important as this update.

 The CentOS infrastructure has hundreds of servers, most of them were
not impacted (as an example). In fact, we seem to have had this happen
on only one machine in those hundreds so far.  It is a problem, obviously.

The issue seems to be with the shim package (not the grub or kernel
packages) and we are currently working with Red Hat on a fix.  This
issue happened in many Linux OSes and even Windows, not just RHEL and
CentOS.

We will push a fix as soon as one is available.

I would hold off on installing this until we release the new fixes.
> 
> In my case, the restore procedure reported by RH in case of reboot does
> not work and reports that the packages are already at the lowest version
> and that the downgrade is not possibile. I don't know why.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20200731/62199254/attachment-0002.sig>

On 7/31/20 11:20 PM, Johnny Hughes wrote:
> On 7/31/20 11:24 AM, Alessandro Baggi wrote:
>>
>> Il 31/07/20 13:08, ja ha scritto:
>>> On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
>>>> I am running an Intel x64 machine using UEFI to boot an SSD.
>>>>
>>>> Installing the latest yum update which includes grub2 and
kernel
>>>> 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable,
blank
>>>> screen where grub should be, no error messages, just hangs.
>>>>
>>>> After some hours I managed to modify another bootable partition
>>>> (containing older software) and boot it from there.
>>>>
>>>> After that, I? found out it is a known problem.
>>>>
>>>> The main point of this message is to make people aware of the
problem
>>>> and suggest admins don't run 'yum update' until
they understand the
>>>> problem and have a fix at hand.
>>>>
>>>> See 'UEFI boot blank screen post update' for a solution
and directions
>>>> to the redhat article.
>>>>
>>>> Regards
>>>>
>>>> Alan
>>>>
>>> I have been punished by this bug - it is/was very nasty.
>>>
>> Me too. Luckily it happened on a test machine.
>>
>> Sorry but seems that those packages were not tested before pushing them
>> in the update repo. Would be great to know what happened to the
>> mainstream chains and how a package like grub reached the update repo
>> when it has serious problem (genuine curiosity but not to blame them).
> 
> Of course it was tested before it was pushed. Obviously this is not a
> problem with every install.  Surely you don't think we push items
> without doing any testing.  Certainly not items as important as this
update.
> 
>   The CentOS infrastructure has hundreds of servers, most of them were
> not impacted (as an example). In fact, we seem to have had this happen
> on only one machine in those hundreds so far.  It is a problem, obviously.
> 
> The issue seems to be with the shim package (not the grub or kernel
> packages) and we are currently working with Red Hat on a fix.  This
> issue happened in many Linux OSes and even Windows, not just RHEL and
> CentOS.
> 
> We will push a fix as soon as one is available.
> 
> I would hold off on installing this until we release the new fixes.
> 
>>
>> In my case, the restore procedure reported by RH in case of reboot does
>> not work and reports that the packages are already at the lowest
version
>> and that the downgrade is not possibile. I don't know why.
> 
It tanked my machine.  I managed to get it to boot into emergency mode 
where I eventually got it to boot on an old kernel as shown in the paste 
below.  After getting it to boot I used grubby to set the default kernel 
to the oldest one on the machine.  Surprisingly enough neither of the 
two most recent kernels would boot even though the machine was running 
on the second oldest kernel when I ran the update that tanked it.  That 
adds credence to the comment above that the problem was not the new 
kernel since it trashed both the new kernel and the one before.  Both 
were from the same series.  The 147 kernel works but neither of newer 
ones would.

CentOS Linux release 8.2.2004 (Core)

enp5s0 IP Address = 192.168.15.131

Linux localhost.localdomain 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 
13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

  Number of cores = 32
  00:55:13 up  8:37,  1 user,  load average: 0.31, 0.48, 0.44

amdgpu-pci-0900
Adapter: PCI adapter
vddgfx:       +0.83 V
fan1:         771 RPM  (min =    0 RPM, max = 3700 RPM)
temp1:        +45.0?C  (crit = +94.0?C, hyst = -273.1?C)
power1:       31.07 W  (cap = 125.00 W)

acpitz-virtual-0
Adapter: Virtual device
temp1:        +16.8?C  (crit = +20.8?C)
temp2:        +16.8?C  (crit = +20.8?C)

amdgpu-pci-0400
Adapter: PCI adapter
vddgfx:       +0.72 V
fan1:         768 RPM  (min =    0 RPM, max = 3700 RPM)
temp1:        +39.0?C  (crit = +94.0?C, hyst = -273.1?C)
power1:       30.10 W  (cap = 125.00 W)



-- 
     _
    ?v?
   /(_)\
    ^ ^  Mark LaPierre
Registered Linux user No #267004
https://linuxcounter.net/
****

On Fri, Jul 31, 2020 at 10:20:28PM -0500, Johnny Hughes wrote:
...
> 
> The issue seems to be with the shim package (not the grub or kernel
> packages) and we are currently working with Red Hat on a fix.  This
> issue happened in many Linux OSes and even Windows, not just RHEL and
> CentOS.
> 
> We will push a fix as soon as one is available.
> 
> I would hold off on installing this until we release the new fixes.
> 
On a system that does not use any of the shim packages, would you
recommend updating the grub2 and mokutil packages or holding off?

Jon
-- 
Jon H. LaBadie                  jcu at labadie.us

Hi Johnny,
thank you very much for clarification.

You said that in the centos infrastructure only one server got the problem.
What are the conditions that permit the breakage? There is a particular
configuration (hw/sw) case that match always the problem or it is random?

Thank you

Il Sab 1 Ago 2020, 05:20 Johnny Hughes <johnny at centos.org> ha scritto:
> On 7/31/20 11:24 AM, Alessandro Baggi wrote:
> >
> > Il 31/07/20 13:08, ja ha scritto:
> >> On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
> >>> I am running an Intel x64 machine using UEFI to boot an SSD.
> >>>
> >>> Installing the latest yum update which includes grub2 and
kernel
> >>> 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable,
blank
> >>> screen where grub should be, no error messages, just hangs.
> >>>
> >>> After some hours I managed to modify another bootable
partition
> >>> (containing older software) and boot it from there.
> >>>
> >>> After that, I  found out it is a known problem.
> >>>
> >>> The main point of this message is to make people aware of the
problem
> >>> and suggest admins don't run 'yum update' until
they understand the
> >>> problem and have a fix at hand.
> >>>
> >>> See 'UEFI boot blank screen post update' for a
solution and directions
> >>> to the redhat article.
> >>>
> >>> Regards
> >>>
> >>> Alan
> >>>
> >> I have been punished by this bug - it is/was very nasty.
> >>
> > Me too. Luckily it happened on a test machine.
> >
> > Sorry but seems that those packages were not tested before pushing
them
> > in the update repo. Would be great to know what happened to the
> > mainstream chains and how a package like grub reached the update repo
> > when it has serious problem (genuine curiosity but not to blame them).
>
> Of course it was tested before it was pushed. Obviously this is not a
> problem with every install.  Surely you don't think we push items
> without doing any testing.  Certainly not items as important as this
> update.
>
>  The CentOS infrastructure has hundreds of servers, most of them were
> not impacted (as an example). In fact, we seem to have had this happen
> on only one machine in those hundreds so far.  It is a problem, obviously.
>
> The issue seems to be with the shim package (not the grub or kernel
> packages) and we are currently working with Red Hat on a fix.  This
> issue happened in many Linux OSes and even Windows, not just RHEL and
> CentOS.
>
> We will push a fix as soon as one is available.
>
> I would hold off on installing this until we release the new fixes.
>
> >
> > In my case, the restore procedure reported by RH in case of reboot
does
> > not work and reports that the packages are already at the lowest
version
> > and that the downgrade is not possibile. I don't know why.
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>