CentOS virt - Jun 2019 - Speculative attack mitigations

Hi folks,


Firstly; apologies in advance for what is a head wrecker of keeping on top of
the speculative mitigations and also if this is a duplicate email; my first copy
didn't seem to make it into the archive.  Also a disclaimer that I may have
misunderstood elements of the below but please bear with me.


I write this hoping to find out a bit more about the state of the relevant
kernel patches for the various speculative attack mitigations in the current
4.9.177-35 kernel used in the centos-virt tree.


For reference; I am running a Dell R620, using an E5-2620 v2 CPU, using
4.9.177-35 from the xen-410 tree in the EL6 branch ; though I am still using xen
4.6 at this time.  I have been told in the irc channel that the kernel is not
built against a specific version of xen so this should make no odds.


As per
https://www.dell.com/support/home/ie/en/iebsdt1/drivers/driversdetails?driverId=NGYCX
- bios version 2.70 (installed) contains mitigations for CVE-2018-3639 (variant
3a route system register read) and CVE-2018-3640 (variant 4, speculative store
bypass).  I am also running the latest available intel microcode for the CPU; as
imported via 'module /GenuineIntel.bin' and 'ucode=2' in my
grub.conf to import (as taken from
https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files).


# uname -r
4.9.177-35.el6.x86_64

# cat /proc/cmdline
ro root=UUID=0e51d8dc-6409-4715-83f4-f86fa24083e2 rd_NO_LUKS  KEYBOARDTYPE=pc
KEYTABLE=uk LANG=en_US.UTF-8 rd_NO_MD nodmraid SYSFONT=latarcyrheb-sun16
crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet intel_idle.max_cstate=0
processor.max_cstate=0 l1ft=full mds=full,nosmt ssbd=force-on

# grep -e flags -e "model name" -e microcode -e bugs /proc/cpuinfo |
sort -u
model name      : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
microcode       : 0x42e
flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr
sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon rep_good nopl nonstop_tsc
pni pclmulqdq monitor est ssse3 cx16 sse4_1 sse4_2 popcnt tsc_deadline_timer aes
xsave avx f16c rdrand hypervisor lahf_lm fsgsbase erms xsaveopt dtherm ida arat
pln pts
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds

You will note that there is a lack of flags for ssbd, spec_ctrl, intel_stibp,
flush_l1d, ibrs, etc.

# for i in /sys/devices/system/cpu/vulnerabilities/* ; do echo -n "$i :
"; cat $i ; done
/sys/devices/system/cpu/vulnerabilities/l1tf : Mitigation: PTE Inversion
/sys/devices/system/cpu/vulnerabilities/mds : Vulnerable: Clear CPU buffers
attempted, no microcode; SMT Host state unknown
/sys/devices/system/cpu/vulnerabilities/meltdown : Vulnerable
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass : Vulnerable
/sys/devices/system/cpu/vulnerabilities/spectre_v1 : Mitigation: __user pointer
sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2 : Mitigation: Full generic
retpoline, STIBP: disabled, RSB filling

Despite having relevant microcode, I am still seeing the box as showing
vulnerable to meltdown, speculative store bypass (ssbd), as well as MDS via
MD_CLEAR mitigation.

When I mount the debugfs, the flag for ibrs is missing - preventing me enabling
it as a mitigation for ssbd (due to lack of relevant cpu flag):

# mount -t debugfs none /sys/kernel/debug/


# ls -lahtr /sys/kernel/debug/x86/ibrs_enabled
ls: cannot access /sys/kernel/debug/x86/ibrs_enabled: No such file or directory


I have other R620's with the same CPUs running stock el6 kernels that are
showing as fully patched to these issues.   Could I please get some feedback
from whoever builds out the kernels; if the mitigations/patches are in place in
the 4.9.177-35 kernel for the various speculative mitigations?

Specifically; CVE-2018-3639, CVE-2018-3640, CVE-2018-3646, CVE-2018-12126,
CVE-2018-12130, CVE-2018-12127 and CVE-2019-11091.


Many thanks for your response.


Regards,

Jonathan

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20190612/d8c2c1b3/attachment-0006.html>

On 6/12/19 3:09 AM, Jonathan Archdale - Blacknight
wrote:
> Hi folks,
> 
> 
> Firstly; apologies in advance for what is a head wrecker of keeping on
> top of the speculative mitigations and also if this is a duplicate
> email; my first copy didn't seem to make it into the archive.? Also a
> disclaimer that I may have misunderstood elements of the below but
> please bear?with me.
> 
> 
> I write this hoping to find out a bit more about the state of the
> relevant kernel patches for the various speculative attack mitigations
> in the current 4.9.177-35 kernel used in the centos-virt tree.? 
4.9.177 should contain all mitigations up to and including md_clear for
MDS, but when running on Xen, you need cooperation from Xen for many of
these fixes as well.
> For reference; I am running a Dell R620, using an E5-2620 v2 CPU, using
> 4.9.177-35 from the xen-410 tree in the EL6 branch ; though I am still
> using xen 4.6 at this time.? I have been told in the irc channel that
> the kernel is not built against a specific version of xen so this should
> make no odds.
This is correct, the same kernel build gets put into all the xen trees
from a "common" repo.
> As
>
per?https://www.dell.com/support/home/ie/en/iebsdt1/drivers/driversdetails?driverId=NGYCX?-
> bios version 2.70 (installed) contains mitigations for?CVE-2018-3639
> (variant 3a route system register read) and?CVE-2018-3640 (variant 4,
> speculative store bypass).? I am also running the latest available intel
> microcode for the CPU; as imported via 'module /GenuineIntel.bin'
and
> 'ucode=2' in my grub.conf to import (as taken
> from?https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files).
> 
> 
> # uname -r
> 4.9.177-35.el6.x86_64
> 
> # cat /proc/cmdline?
> ro root=UUID=0e51d8dc-6409-4715-83f4-f86fa24083e2 rd_NO_LUKS?
> KEYBOARDTYPE=pc KEYTABLE=uk LANG=en_US.UTF-8 rd_NO_MD nodmraid
> SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_NO_LVM rd_NO_DM rhgb quiet
> intel_idle.max_cstate=0 processor.max_cstate=0 l1ft=full mds=full,nosmt
> ssbd=force-on
> 
> # grep -e flags -e "model name" -e microcode -e
bugs?/proc/cpuinfo | sort -u
> model name? ? ? : Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
> microcode? ? ? ?: 0x42e
> flags: fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx
> fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon rep_good
> nopl nonstop_tsc pni pclmulqdq monitor est ssse3 cx16 sse4_1 sse4_2
> popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm
> fsgsbase erms xsaveopt dtherm ida arat pln pts
> bugs? ? ? ? ? ? : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
> l1tf mds
>
> You will note that there is a lack of flags for ssbd,
> spec_ctrl,?intel_stibp,?flush_l1d, ibrs, etc.??
Your microcode is correct, per:

https://www.intel.com/content/dam/www/public/us/en/documents/corporate-information/SA00233-microcode-update-guidance_05132019.pdf

The kernel however is not picking up any of the expected microcode
features one would expect in the flags list.

I have a E5-2609 v2 that uses the same microcode, but mine shows the
correct flags in the list.  The key difference between our environments
is that you are running your kernel on top of Xen 4.6 which is EOL since
Oct 2018 upstream.  The CentOS builds were never updated past 4.6.6-12,
which was released in May 2018, so it doesn't include the fixes for L1TF
or SSBD either.

You didn't include any details of the Xen packages you're running, but
make sure you are at least on Xen 4.6.6-12.  'xl info' may be useful
here.

That said, I can't explain why you don't see the ibrs or ibpb flags in
your list.

As a possible useful exercise, take a look at 'xl dmesg' for messages
containing "microcode:" as well as a section labeled "Speculative
mitigation facilities:" to verify that Xen is recognizing and supporting
the necessary CPU features.
> # for i in /sys/devices/system/cpu/vulnerabilities/* ; do echo -n "$i
:
> "; cat $i ; done
> /sys/devices/system/cpu/vulnerabilities/l1tf : Mitigation: PTE Inversion
> /sys/devices/system/cpu/vulnerabilities/mds : Vulnerable: Clear CPU
> buffers attempted, no microcode; SMT Host state unknown
> /sys/devices/system/cpu/vulnerabilities/meltdown : Vulnerable
> /sys/devices/system/cpu/vulnerabilities/spec_store_bypass : Vulnerable
> /sys/devices/system/cpu/vulnerabilities/spectre_v1 : Mitigation: __user
> pointer sanitization
> /sys/devices/system/cpu/vulnerabilities/spectre_v2 : Mitigation: Full
> generic retpoline, STIBP: disabled, RSB filling
> 
> Despite having relevant microcode, I am still seeing the box as showing
> vulnerable to meltdown, speculative store bypass (ssbd), as well as MDS
> via MD_CLEAR mitigation.
Running inside of Xen dom0 is not quite the same as running natively on
your hardware.  Your dom0 is a type of guest, so the mitigations
described in the /sys/ location only reflect the state of the dom0
itself, not Xen itself or other guests.

The meltdown section will always show as vulnerable because the Linux
kernel does not recognize the Xen-based mitigation, even though it is
effective for dom0.  Also, PV domains running inside a Xen release
patched for meltdown are protected in spite of the indicator by the
guest kernel.
> When I mount the debugfs, the flag for ibrs is missing - preventing me
> enabling it as a mitigation for ssbd (due to lack of relevant cpu flag):
> 
> # mount -t debugfs none /sys/kernel/debug/
> 
> 
> # ls -lahtr /sys/kernel/debug/x86/ibrs_enabled
> ls: cannot access /sys/kernel/debug/x86/ibrs_enabled: No such file or
> directory
> 
> I have other R620's with the same CPUs?running stock el6 kernels that
> are showing as fully patched to these issues.? ?Could I please get some
> feedback from whoever builds out the kernels; if the mitigations/patches
> are in place in the 4.9.177-35 kernel for the various speculative
> mitigations?
> 
>
Specifically;?CVE-2018-3639,?CVE-2018-3640,?CVE-2018-3646,?CVE-2018-12126,?CVE-2018-12130,?CVE-2018-12127
> and?CVE-2019-11091.
Just to reiterate, these fixes are in 4.9.177 but the mitigations
generally require cooperation from Xen, and Xen 4.6 builds from CentOS
do not contain fixes for disclosed issues from May 2018 onward.

For proper mitigation, you need to upgrade to Xen 4.8 or newer, but I
would suggest you consider going to 4.10 if possible (or 4.12 if you
upgrade to CentOS 7) to give you more headroom before EOL as well as a
number of performance improvements, both related to the mitigations and
otherwise.

-- 
Kevin Stange
Chief Technology Officer
Steadfast | Managed Infrastructure, Datacenter and Cloud Services
800 S Wells, Suite 190 | Chicago, IL 60607
312.602.2689 X203 | Fax: 312.602.2688
kevin at steadfast.net | www.steadfast.net