CentOS - Jan 2018 - /lib/firmware/microcode.dat update on CentOS 6

> Look at:
> 
> https://t.co/6fT61xgtGH
> 
> Get the latest microcode.dat file from here:
> 
> https://t.co/zPwagbeJFY
> 
> See how to update the microcode from the links at the bottom of this page:
> 
> https://t.co/EOgclWdHCw
> 
> An before anyone asks .. I have no idea why Red Hat chose this path,
> they did.  It doesn't matter if I (or anyone else) agrees with the
> decision.  It is what it is.
> 
**I'm not blaming you.**

But can I just clarify. We have to *manually* install the microcode
update an EL7 in order to be protected against Spectre? EL6 as well?

Presumably this is to remove RH from the loop and to stop people
blaming them - i.e. this is between Intel and the customer, it's
nothing to do with them.

What about future microcode updates? They come out reasonably regularly
 (2 or 3 times a year) - are RH going to absolve themselves from all
future updates because presumably the next update will also contain the
Spectre fixes?

So, before I re-invent the wheel, does anyone have automation scripts
to do the microcode update? I don't relish the prospect of doing this
manually on a couple of hundred machines. Is it reasonable to grab the
microcode_ctl SRPM and create my own updated RPM to do it?

P.

On 01/18/2018 03:41 AM, Pete Biggs wrote:
> 
>> Look at:
>>
>> https://t.co/6fT61xgtGH
>>
>> Get the latest microcode.dat file from here:
>>
>> https://t.co/zPwagbeJFY
>>
>> See how to update the microcode from the links at the bottom of this
page:
>>
>> https://t.co/EOgclWdHCw
>>
>> An before anyone asks .. I have no idea why Red Hat chose this path,
>> they did.  It doesn't matter if I (or anyone else) agrees with the
>> decision.  It is what it is.
>>
> **I'm not blaming you.**
> 
> But can I just clarify. We have to *manually* install the microcode
> update an EL7 in order to be protected against Spectre? EL6 as well?
> 
> Presumably this is to remove RH from the loop and to stop people
> blaming them - i.e. this is between Intel and the customer, it's
> nothing to do with them.
> 
No, this is because at least one major CPU (Intel type 79) is completely
broken by the Intel Microcode Update.  Those machines can't boot after
the microcode rpm is installed.  It impacts at least these processors:

Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz
Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.50GHz

There may be others.

https://bugzilla.redhat.com/show_bug.cgi?id=1532283

That means, it is NOT full-proof install and likely leaves many
servers/machines broken.  I suppose that the decision is, go pack to
what works for all machines (the last known good install) and let admins
work with their hardware vendor because the alternative is breaking things.

This also needs to be fixed with a combination of firmware updates AND
microcode updates.  All of that is outside the expertise of a Linux
vendor and is unique for each processor, chipset and firmware combination.
> What about future microcode updates? They come out reasonably regularly
>  (2 or 3 times a year) - are RH going to absolve themselves from all
> future updates because presumably the next update will also contain the
> Spectre fixes?
> 

How they handle it in the future I have no way of knowing, but if you
had 20,000 servers with the impacted CPU and you updated and could not
reboot, I would assume that you did not appreciate it.
> So, before I re-invent the wheel, does anyone have automation scripts
> to do the microcode update? I don't relish the prospect of doing this
> manually on a couple of hundred machines. Is it reasonable to grab the
> microcode_ctl SRPM and create my own updated RPM to do it?
> 
That is what I have found so far with a bit of research.

This is NOTHING about who to blame and everything about stable, working
updates .. at least it seems so to me.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20180118/cda9be29/attachment-0001.sig>

On Thu, Jan 18, 2018 at 5:03 AM, Johnny Hughes <johnny at centos.org>
wrote:
> On 01/18/2018 03:41 AM, Pete Biggs wrote:
> >
> >> Look at:
> >>
> >> https://t.co/6fT61xgtGH
> >>
> >> Get the latest microcode.dat file from here:
> >>
> >> https://t.co/zPwagbeJFY
> >>
> >> See how to update the microcode from the links at the bottom of
this
> page:
> >>
> >> https://t.co/EOgclWdHCw
> >>
> >> An before anyone asks .. I have no idea why Red Hat chose this
path,
> >> they did.  It doesn't matter if I (or anyone else) agrees with
the
> >> decision.  It is what it is.
> >>
> > **I'm not blaming you.**
> >
> > But can I just clarify. We have to *manually* install the microcode
> > update an EL7 in order to be protected against Spectre? EL6 as well?
> >
> > Presumably this is to remove RH from the loop and to stop people
> > blaming them - i.e. this is between Intel and the customer, it's
> > nothing to do with them.
> >
>
> No, this is because at least one major CPU (Intel type 79) is completely
> broken by the Intel Microcode Update.  Those machines can't boot after
> the microcode rpm is installed.  It impacts at least these processors:
>
> Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
> Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz
> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.50GHz
>
> There may be others.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1532283
>
> That means, it is NOT full-proof install and likely leaves many
> servers/machines broken.  I suppose that the decision is, go pack to
> what works for all machines (the last known good install) and let admins
> work with their hardware vendor because the alternative is breaking things.
>
> This also needs to be fixed with a combination of firmware updates AND
> microcode updates.  All of that is outside the expertise of a Linux
> vendor and is unique for each processor, chipset and firmware combination.
>
> > What about future microcode updates? They come out reasonably
regularly
> >  (2 or 3 times a year) - are RH going to absolve themselves from all
> > future updates because presumably the next update will also contain
the
> > Spectre fixes?
> >
>
>
> How they handle it in the future I have no way of knowing, but if you
> had 20,000 servers with the impacted CPU and you updated and could not
> reboot, I would assume that you did not appreciate it.
>
> > So, before I re-invent the wheel, does anyone have automation scripts
> > to do the microcode update? I don't relish the prospect of doing
this
> > manually on a couple of hundred machines. Is it reasonable to grab the
> > microcode_ctl SRPM and create my own updated RPM to do it?
> >
>
> That is what I have found so far with a bit of research.
>
> This is NOTHING about who to blame and everything about stable, working
> updates .. at least it seems so to me.
>
>
So, if we applied the previous microcode update, and all our machines
rebooted OK, then we don't need to fallback?

Also, do we know if the updated CentOS microcode RPM reverted the microcode
for *all* Intel CPUs, or just the ones that had issues? In other words, if
I apply the latest microcode update to our 100+ machines (which all have
the previous update, and are OK) will they revert to a vulnerable state?


-- 
Matt Phelps
System Administrator, Computation Facility
Harvard - Smithsonian Center for Astrophysics
mphelps at cfa.harvard.edu, http://www.cfa.harvard.edu

On 01/18/18 03:41, Pete Biggs wrote:
> 
>> Look at:
>>
>> https://t.co/6fT61xgtGH
>>
>> Get the latest microcode.dat file from here:
>>
>> https://t.co/zPwagbeJFY
>>
>> See how to update the microcode from the links at the bottom of this
page:
>>
>> https://t.co/EOgclWdHCw
>>
>> An before anyone asks .. I have no idea why Red Hat chose this path,
>> they did.  It doesn't matter if I (or anyone else) agrees with the
>> decision.  It is what it is.
>>
> **I'm not blaming you.**
> 
> But can I just clarify. We have to *manually* install the microcode
> update an EL7 in order to be protected against Spectre? EL6 as well?
> 
> Presumably this is to remove RH from the loop and to stop people
> blaming them - i.e. this is between Intel and the customer, it's
> nothing to do with them.
I bet you are right. I was going to rant about that... then it occurred 
to me that class action against Intel (didn't hear about AMD though) is 
quite likely, so, indeed, RedHat does not want to be even mentioned in 
it, which will be unfair, especially after RedHat putting effort into 
fixing somebody's else crap.

Valeri
> 
> What about future microcode updates? They come out reasonably regularly
>   (2 or 3 times a year) - are RH going to absolve themselves from all
> future updates because presumably the next update will also contain the
> Spectre fixes?
> 
> So, before I re-invent the wheel, does anyone have automation scripts
> to do the microcode update? I don't relish the prospect of doing this
> manually on a couple of hundred machines. Is it reasonable to grab the
> microcode_ctl SRPM and create my own updated RPM to do it?
> 
> P.
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
> 
-- 
++++++++++++++++++++++++++++++++++++++++
Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247
++++++++++++++++++++++++++++++++++++++++

On Thu, 18 Jan 2018 04:03:48 -0600
Johnny Hughes <johnny at centos.org> wrote:
> On 01/18/2018 03:41 AM, Pete Biggs wrote:
> >   
> >> Look at:
> >>
> >> https://t.co/6fT61xgtGH
> >>
> >> Get the latest microcode.dat file from here:
> >>
> >> https://t.co/zPwagbeJFY
> >>
> >> See how to update the microcode from the links at the bottom of
> >> this page:
> >>
> >> https://t.co/EOgclWdHCw
> >>
> >> An before anyone asks .. I have no idea why Red Hat chose this
> >> path, they did.  It doesn't matter if I (or anyone else)
agrees
> >> with the decision.  It is what it is.
> >>  
> > **I'm not blaming you.**
> > 
> > But can I just clarify. We have to *manually* install the microcode
> > update an EL7 in order to be protected against Spectre? EL6 as well?
> > 
> > Presumably this is to remove RH from the loop and to stop people
> > blaming them - i.e. this is between Intel and the customer, it's
> > nothing to do with them.
> >   
> 
> No, this is because at least one major CPU (Intel type 79) is
> completely broken by the Intel Microcode Update.  Those machines
> can't boot after the microcode rpm is installed.  It impacts at least
> these processors:
> 
> Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
> Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz
> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.50GHz
> 
> There may be others.
As a data point, we have the updated microcode running on 600+ Haswell
servers and so far no indication of problems.

We'll keep the ibrs/spectre mitigation this gives us and not revert
(unless it turns out it does cause problems).

/Peter
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20180118/e770e9a8/attachment-0001.sig>

On 01/18/2018 09:42 AM, Valeri Galtsev wrote:
> 
> 
> On 01/18/18 03:41, Pete Biggs wrote:
>>
>>> Look at:
>>>
>>> https://t.co/6fT61xgtGH
>>>
>>> Get the latest microcode.dat file from here:
>>>
>>> https://t.co/zPwagbeJFY
>>>
>>> See how to update the microcode from the links at the bottom of
this
>>> page:
>>>
>>> https://t.co/EOgclWdHCw
>>>
>>> An before anyone asks .. I have no idea why Red Hat chose this
path,
>>> they did.? It doesn't matter if I (or anyone else) agrees with
the
>>> decision.? It is what it is.
>>>
>> **I'm not blaming you.**
>>
>> But can I just clarify. We have to *manually* install the microcode
>> update an EL7 in order to be protected against Spectre? EL6 as well?
>>
>> Presumably this is to remove RH from the loop and to stop people
>> blaming them - i.e. this is between Intel and the customer, it's
>> nothing to do with them.
> 
> I bet you are right. I was going to rant about that... then it occurred
> to me that class action against Intel (didn't hear about AMD though) is
> quite likely, so, indeed, RedHat does not want to be even mentioned in
> it, which will be unfair, especially after RedHat putting effort into
> fixing somebody's else crap.
> 
It isn't about washing hands, lawsuits, or soeoen else's stuff.  It is
about broken microcode updates.

The code from intel was broken .. causing several CPUs not to boot after
update.  That (and only that) is why they were pulled.

Users MUST individually QA the microcode for their individual CPUs, OEM
Frimware, chipset etc.

The issue here is that the microcode broke peoples machines .. therefore
it had to be rolled back.  All the other discussion is full and total BS.

It is likely ONCE all the microcode updates are tested and completely
working that Red Hat will again include it in the microcode_ctl RPM ..
but that can't put stuff in there that is breaking machines.

While things are beuing released as QA quality, they are going to have
to be done individually by admins .. that's just how it is.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20180122/a1fce928/attachment-0001.sig>