CentOS - Aug 2020 - 8.2.2004 Latest yum update renders machine unbootable

On 8/1/20 6:56 AM, david wrote:
> At 02:54 AM 8/1/2020, Alessandro Baggi wrote:
>> Hi Johnny,
>> thank you very much for clarification.
>>
>> You said that in the centos infrastructure only one server got the 
>> problem.
>> What are the conditions that permit the breakage? There is a particular
>> configuration (hw/sw) case that match always the problem or it is 
>> random?
>>
>> Thank you
>
> I have two servers running Centos 7 on apple hardware (one mac-mini 
> and one mac server).? They both failed to reboot a few days ago.? So 
> perhaps whatever anti-boot bug hit Centos 8, also hit Centos 7.? I 
> can't tell what version got updated since the system simply fails to 
> boot.? I don't even get a grub screen. I'll have to rebuild the 
> systems from scratch.
>
>
You should be able to boot off of installation media into rescue mode, 
and downgrade the grub2* and/or shim* RPMs.

-Greg

At 01:03 PM 8/1/2020, you wrote:
>On 8/1/20 6:56 AM, david wrote:
>>At 02:54 AM 8/1/2020, Alessandro Baggi wrote:
>>>Hi Johnny,
>>>thank you very much for clarification.
>>>
>>>You said that in the centos infrastructure only one server got the
problem.
>>>What are the conditions that permit the breakage? There is a
particular
>>>configuration (hw/sw) case that match always the problem or it is
random?
>>>
>>>Thank you
>>
>>I have two servers running Centos 7 on apple 
>>hardware (one mac-mini and one mac 
>>server).?  They both failed to reboot a few 
>>days ago.?  So perhaps whatever anti-boot bug 
>>hit Centos 8, also hit Centos 7.?  I can't tell 
>>what version got updated since the system 
>>simply fails to boot.?  I don't even get a grub 
>>screen. I'll have to rebuild the systems from scratch.
>>
>
>You should be able to boot off of installation 
>media into rescue mode, and downgrade the grub2* and/or shim* RPMs.
>
>-Greg
This is a good idea, if I knew how to 
"downgrade...".  But in any event, I had decided 
to rebuild from scratch, which of course failed 
as soon as I did an yum update.  So, I'm 
installing 7.8.2003 with no updates until I see 
the "all clear -- updates will no longer make 
your system unbootable" message from the Centos team.

In my many years of blindly updating my 
installations, starting from the free Redhat 
distributions, through Whitehat and onto Centos, 
this is the first disaster, and luckily, it 
didn't hit all my systems.  Let's hope there aren't many more.

David

Don't forget that EL 7 (non-UEFI systems) & 8 support booting from LVM
snapshot (a.k.a BOOM Boot Manager) , so revert is 2  steps away:
- boot from the snapshot (grub menu)
- revert from the lvm snapshot
- reboot  and wait for the revert to complete


Best Regards,
Strahil Nikolov

?? 1 ?????? 2020 ?. 23:58:27 GMT+03:00, david <david at daku.org>
??????:
>At 01:03 PM 8/1/2020, you wrote:
>>On 8/1/20 6:56 AM, david wrote:
>>>At 02:54 AM 8/1/2020, Alessandro Baggi wrote:
>>>>Hi Johnny,
>>>>thank you very much for clarification.
>>>>
>>>>You said that in the centos infrastructure only one server got
the
>problem.
>>>>What are the conditions that permit the breakage? There is a
>particular
>>>>configuration (hw/sw) case that match always the problem or it
is
>random?
>>>>
>>>>Thank you
>>>
>>>I have two servers running Centos 7 on apple 
>>>hardware (one mac-mini and one mac 
>>>server).?  They both failed to reboot a few 
>>>days ago.?  So perhaps whatever anti-boot bug 
>>>hit Centos 8, also hit Centos 7.?  I can't tell 
>>>what version got updated since the system 
>>>simply fails to boot.?  I don't even get a grub 
>>>screen. I'll have to rebuild the systems from scratch.
>>>
>>
>>You should be able to boot off of installation 
>>media into rescue mode, and downgrade the grub2* and/or shim* RPMs.
>>
>>-Greg
>
>This is a good idea, if I knew how to 
>"downgrade...".  But in any event, I had decided 
>to rebuild from scratch, which of course failed 
>as soon as I did an yum update.  So, I'm 
>installing 7.8.2003 with no updates until I see 
>the "all clear -- updates will no longer make 
>your system unbootable" message from the Centos team.
>
>In my many years of blindly updating my 
>installations, starting from the free Redhat 
>distributions, through Whitehat and onto Centos, 
>this is the first disaster, and luckily, it 
>didn't hit all my systems.  Let's hope there aren't many more.
>
>David 
>
>_______________________________________________
>CentOS mailing list
>CentOS at centos.org
>https://lists.centos.org/mailman/listinfo/centos

Il 01/08/20 22:03, Greg Bailey ha scritto:
> On 8/1/20 6:56 AM, david wrote:
>> At 02:54 AM 8/1/2020, Alessandro Baggi wrote:
>>> Hi Johnny,
>>> thank you very much for clarification.
>>>
>>> You said that in the centos infrastructure only one server got the 
>>> problem.
>>> What are the conditions that permit the breakage? There is a
particular
>>> configuration (hw/sw) case that match always the problem or it is 
>>> random?
>>>
>>> Thank you
>>
>> I have two servers running Centos 7 on apple hardware (one mac-mini 
>> and one mac server).? They both failed to reboot a few days ago.? So 
>> perhaps whatever anti-boot bug hit Centos 8, also hit Centos 7.? I 
>> can't tell what version got updated since the system simply fails
to
>> boot.? I don't even get a grub screen. I'll have to rebuild the
>> systems from scratch.
>>
>>
>
> You should be able to boot off of installation media into rescue mode, 
> and downgrade the grub2* and/or shim* RPMs.
>
> -Greg
>
I did the downgrade on a fresh install of c8.2 but yum said that all 
selected packages (grub2,shim...) are already to the lowest version and 
the downgrade is not possibile, ending with "nothing to do".

On 8/2/20 2:04 AM, Alessandro Baggi wrote:
> 
> Il 01/08/20 22:03, Greg Bailey ha scritto:
>> On 8/1/20 6:56 AM, david wrote:
>>> At 02:54 AM 8/1/2020, Alessandro Baggi wrote:
>>>> Hi Johnny,
>>>> thank you very much for clarification.
>>>>
>>>> You said that in the centos infrastructure only one server got
the
>>>> problem.
>>>> What are the conditions that permit the breakage? There is a
particular
>>>> configuration (hw/sw) case that match always the problem or it
is
>>>> random?
>>>>
>>>> Thank you
>>>
>>> I have two servers running Centos 7 on apple hardware (one mac-mini
>>> and one mac server).? They both failed to reboot a few days ago.?
So
>>> perhaps whatever anti-boot bug hit Centos 8, also hit Centos 7.? I
>>> can't tell what version got updated since the system simply
fails to
>>> boot.? I don't even get a grub screen. I'll have to rebuild
the
>>> systems from scratch.
>>>
>>>
>>
>> You should be able to boot off of installation media into rescue mode,
>> and downgrade the grub2* and/or shim* RPMs.
>>
>> -Greg
>>
> I did the downgrade on a fresh install of c8.2 but yum said that all
> selected packages (grub2,shim...) are already to the lowest version and
> the downgrade is not possibile, ending with "nothing to do".
Ok .. We are running through some final testing now for CentOS Linux 8
and CentOS Stream .. updates later today for EL8.

For CentOS Linux 7 .. I just pushed the latest shim packages (we had to
get these signed by Microsoft .. as do all distros that do shim.
Microsoft is the official CA for secureboot.

So in the next few hours, after the mirrors sync up .. you should be
able to fix any EL7 machines.

I'll post here again once we have pushed the EL8 and CentOS Stream updates.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20200802/36f59c5b/attachment.sig>