CentOS - Jan 2018 - /lib/firmware/microcode.dat update on CentOS 6

On Thu, Jan 18, 2018 at 5:03 AM, Johnny Hughes <johnny at centos.org>
wrote:
> On 01/18/2018 03:41 AM, Pete Biggs wrote:
> >
> >> Look at:
> >>
> >> https://t.co/6fT61xgtGH
> >>
> >> Get the latest microcode.dat file from here:
> >>
> >> https://t.co/zPwagbeJFY
> >>
> >> See how to update the microcode from the links at the bottom of
this
> page:
> >>
> >> https://t.co/EOgclWdHCw
> >>
> >> An before anyone asks .. I have no idea why Red Hat chose this
path,
> >> they did.  It doesn't matter if I (or anyone else) agrees with
the
> >> decision.  It is what it is.
> >>
> > **I'm not blaming you.**
> >
> > But can I just clarify. We have to *manually* install the microcode
> > update an EL7 in order to be protected against Spectre? EL6 as well?
> >
> > Presumably this is to remove RH from the loop and to stop people
> > blaming them - i.e. this is between Intel and the customer, it's
> > nothing to do with them.
> >
>
> No, this is because at least one major CPU (Intel type 79) is completely
> broken by the Intel Microcode Update.  Those machines can't boot after
> the microcode rpm is installed.  It impacts at least these processors:
>
> Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
> Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz
> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.50GHz
>
> There may be others.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1532283
>
> That means, it is NOT full-proof install and likely leaves many
> servers/machines broken.  I suppose that the decision is, go pack to
> what works for all machines (the last known good install) and let admins
> work with their hardware vendor because the alternative is breaking things.
>
> This also needs to be fixed with a combination of firmware updates AND
> microcode updates.  All of that is outside the expertise of a Linux
> vendor and is unique for each processor, chipset and firmware combination.
>
> > What about future microcode updates? They come out reasonably
regularly
> >  (2 or 3 times a year) - are RH going to absolve themselves from all
> > future updates because presumably the next update will also contain
the
> > Spectre fixes?
> >
>
>
> How they handle it in the future I have no way of knowing, but if you
> had 20,000 servers with the impacted CPU and you updated and could not
> reboot, I would assume that you did not appreciate it.
>
> > So, before I re-invent the wheel, does anyone have automation scripts
> > to do the microcode update? I don't relish the prospect of doing
this
> > manually on a couple of hundred machines. Is it reasonable to grab the
> > microcode_ctl SRPM and create my own updated RPM to do it?
> >
>
> That is what I have found so far with a bit of research.
>
> This is NOTHING about who to blame and everything about stable, working
> updates .. at least it seems so to me.
>
>
So, if we applied the previous microcode update, and all our machines
rebooted OK, then we don't need to fallback?

Also, do we know if the updated CentOS microcode RPM reverted the microcode
for *all* Intel CPUs, or just the ones that had issues? In other words, if
I apply the latest microcode update to our 100+ machines (which all have
the previous update, and are OK) will they revert to a vulnerable state?


-- 
Matt Phelps
System Administrator, Computation Facility
Harvard - Smithsonian Center for Astrophysics
mphelps at cfa.harvard.edu, http://www.cfa.harvard.edu

On 01/18/2018 07:51 AM, Phelps, Matthew wrote:
> On Thu, Jan 18, 2018 at 5:03 AM, Johnny Hughes <johnny at centos.org>
wrote:
> 
>> On 01/18/2018 03:41 AM, Pete Biggs wrote:
>>>
>>>> Look at:
>>>>
>>>> https://t.co/6fT61xgtGH
>>>>
>>>> Get the latest microcode.dat file from here:
>>>>
>>>> https://t.co/zPwagbeJFY
>>>>
>>>> See how to update the microcode from the links at the bottom of
this
>> page:
>>>>
>>>> https://t.co/EOgclWdHCw
>>>>
>>>> An before anyone asks .. I have no idea why Red Hat chose this
path,
>>>> they did.  It doesn't matter if I (or anyone else) agrees
with the
>>>> decision.  It is what it is.
>>>>
>>> **I'm not blaming you.**
>>>
>>> But can I just clarify. We have to *manually* install the microcode
>>> update an EL7 in order to be protected against Spectre? EL6 as
well?
>>>
>>> Presumably this is to remove RH from the loop and to stop people
>>> blaming them - i.e. this is between Intel and the customer,
it's
>>> nothing to do with them.
>>>
>>
>> No, this is because at least one major CPU (Intel type 79) is
completely
>> broken by the Intel Microcode Update.  Those machines can't boot
after
>> the microcode rpm is installed.  It impacts at least these processors:
>>
>> Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
>> Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz
>> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
>> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.50GHz
>>
>> There may be others.
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1532283
>>
>> That means, it is NOT full-proof install and likely leaves many
>> servers/machines broken.  I suppose that the decision is, go pack to
>> what works for all machines (the last known good install) and let
admins
>> work with their hardware vendor because the alternative is breaking
things.
>>
>> This also needs to be fixed with a combination of firmware updates AND
>> microcode updates.  All of that is outside the expertise of a Linux
>> vendor and is unique for each processor, chipset and firmware
combination.
>>
>>> What about future microcode updates? They come out reasonably
regularly
>>>  (2 or 3 times a year) - are RH going to absolve themselves from
all
>>> future updates because presumably the next update will also contain
the
>>> Spectre fixes?
>>>
>>
>>
>> How they handle it in the future I have no way of knowing, but if you
>> had 20,000 servers with the impacted CPU and you updated and could not
>> reboot, I would assume that you did not appreciate it.
>>
>>> So, before I re-invent the wheel, does anyone have automation
scripts
>>> to do the microcode update? I don't relish the prospect of
doing this
>>> manually on a couple of hundred machines. Is it reasonable to grab
the
>>> microcode_ctl SRPM and create my own updated RPM to do it?
>>>
>>
>> That is what I have found so far with a bit of research.
>>
>> This is NOTHING about who to blame and everything about stable, working
>> updates .. at least it seems so to me.
>>
>>
> So, if we applied the previous microcode update, and all our machines
> rebooted OK, then we don't need to fallback?
> 
> Also, do we know if the updated CentOS microcode RPM reverted the microcode
> for *all* Intel CPUs, or just the ones that had issues? In other words, if
> I apply the latest microcode update to our 100+ machines (which all have
> the previous update, and are OK) will they revert to a vulnerable state?
> 
> 
It reverted for all .. but, your machines may or may not be protected as
only a subset of machines were updated with the original microcode from
Intel.

It is your call as to what you install .. but the correct method is to
install the current microcode_ctl .. and then research your specific
machine, its CPU, chipset, firmware .. go to the vendor and make sure
you get all the things necessary to mitigate the issues.  It will be
different for each CPU vendor (Intel or AMD), each CPU / Chipset combo,
and even each vendor (Dell may have new firmware for x and y but not z
models, etc.)

There is no one size fits all update for this issue.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20180118/37f4b86b/attachment-0001.sig>

On Thu, Jan 18, 2018 at 9:01 AM, Johnny Hughes <johnny at centos.org>
wrote:
> On 01/18/2018 07:51 AM, Phelps, Matthew wrote:
> > On Thu, Jan 18, 2018 at 5:03 AM, Johnny Hughes <johnny at
centos.org>
> wrote:
> >
> >> On 01/18/2018 03:41 AM, Pete Biggs wrote:
> >>>
> >>>> Look at:
> >>>>
> >>>> https://t.co/6fT61xgtGH
> >>>>
> >>>> Get the latest microcode.dat file from here:
> >>>>
> >>>> https://t.co/zPwagbeJFY
> >>>>
> >>>> See how to update the microcode from the links at the
bottom of this
> >> page:
> >>>>
> >>>> https://t.co/EOgclWdHCw
> >>>>
> >>>> An before anyone asks .. I have no idea why Red Hat chose
this path,
> >>>> they did.  It doesn't matter if I (or anyone else)
agrees with the
> >>>> decision.  It is what it is.
> >>>>
> >>> **I'm not blaming you.**
> >>>
> >>> But can I just clarify. We have to *manually* install the
microcode
> >>> update an EL7 in order to be protected against Spectre? EL6 as
well?
> >>>
> >>> Presumably this is to remove RH from the loop and to stop
people
> >>> blaming them - i.e. this is between Intel and the customer,
it's
> >>> nothing to do with them.
> >>>
> >>
> >> No, this is because at least one major CPU (Intel type 79) is
completely
> >> broken by the Intel Microcode Update.  Those machines can't
boot after
> >> the microcode rpm is installed.  It impacts at least these
processors:
> >>
> >> Intel(R) Xeon(R) CPU E5-2637 v4 @ 3.50GHz
> >> Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz
> >> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz
> >> Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.50GHz
> >>
> >> There may be others.
> >>
> >> https://bugzilla.redhat.com/show_bug.cgi?id=1532283
> >>
> >> That means, it is NOT full-proof install and likely leaves many
> >> servers/machines broken.  I suppose that the decision is, go pack
to
> >> what works for all machines (the last known good install) and let
admins
> >> work with their hardware vendor because the alternative is
breaking
> things.
> >>
> >> This also needs to be fixed with a combination of firmware updates
AND
> >> microcode updates.  All of that is outside the expertise of a
Linux
> >> vendor and is unique for each processor, chipset and firmware
> combination.
> >>
> >>> What about future microcode updates? They come out reasonably
regularly
> >>>  (2 or 3 times a year) - are RH going to absolve themselves
from all
> >>> future updates because presumably the next update will also
contain the
> >>> Spectre fixes?
> >>>
> >>
> >>
> >> How they handle it in the future I have no way of knowing, but if
you
> >> had 20,000 servers with the impacted CPU and you updated and could
not
> >> reboot, I would assume that you did not appreciate it.
> >>
> >>> So, before I re-invent the wheel, does anyone have automation
scripts
> >>> to do the microcode update? I don't relish the prospect of
doing this
> >>> manually on a couple of hundred machines. Is it reasonable to
grab the
> >>> microcode_ctl SRPM and create my own updated RPM to do it?
> >>>
> >>
> >> That is what I have found so far with a bit of research.
> >>
> >> This is NOTHING about who to blame and everything about stable,
working
> >> updates .. at least it seems so to me.
> >>
> >>
> > So, if we applied the previous microcode update, and all our machines
> > rebooted OK, then we don't need to fallback?
> >
> > Also, do we know if the updated CentOS microcode RPM reverted the
> microcode
> > for *all* Intel CPUs, or just the ones that had issues? In other
words,
> if
> > I apply the latest microcode update to our 100+ machines (which all
have
> > the previous update, and are OK) will they revert to a vulnerable
state?
> >
> >
>
> It reverted for all .. but, your machines may or may not be protected as
> only a subset of machines were updated with the original microcode from
> Intel.
>
> It is your call as to what you install .. but the correct method is to
> install the current microcode_ctl .. and then research your specific
> machine, its CPU, chipset, firmware .. go to the vendor and make sure
> you get all the things necessary to mitigate the issues.  It will be
> different for each CPU vendor (Intel or AMD), each CPU / Chipset combo,
> and even each vendor (Dell may have new firmware for x and y but not z
> models, etc.)
>
> There is no one size fits all update for this issue.
>
>
Johnny,

Thank you. We all appreciate your efforts, and your guidance.


-- 
Matt Phelps
System Administrator, Computation Facility
Harvard - Smithsonian Center for Astrophysics
mphelps at cfa.harvard.edu, http://www.cfa.harvard.edu

On 01/18/18 09:01, Johnny Hughes wrote:
> On 01/18/2018 07:51 AM, Phelps, Matthew wrote:
>> On Thu, Jan 18, 2018 at 5:03 AM, Johnny Hughes <johnny at
centos.org> wrote:
>>
>> So, if we applied the previous microcode update, and all our machines
>> rebooted OK, then we don't need to fallback?
>>
>> Also, do we know if the updated CentOS microcode RPM reverted the
microcode
>> for *all* Intel CPUs, or just the ones that had issues? In other words,
if
>> I apply the latest microcode update to our 100+ machines (which all
have
>> the previous update, and are OK) will they revert to a vulnerable
state?
>>
>>
> It reverted for all .. but, your machines may or may not be protected as
> only a subset of machines were updated with the original microcode from
> Intel.
>
> It is your call as to what you install .. but the correct method is to
> install the current microcode_ctl .. and then research your specific
> machine, its CPU, chipset, firmware .. go to the vendor and make sure
> you get all the things necessary to mitigate the issues.  It will be
> different for each CPU vendor (Intel or AMD), each CPU / Chipset combo,
> and even each vendor (Dell may have new firmware for x and y but not z
> models, etc.)
>
> There is no one size fits all update for this issue.
>
OK, so color me confused about the timing in all this.

Do we update the microcode now or do we wait until the latest 
microcode_ctl rpm is available and then tackle this issue?

-- 
Unencumbered by the thought process.
  -- Click and Clack the Tappet brothers


-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.