Xen devel - Jul 2013 - PROPOSAL: Microcode loading under x86 - various options, discussion, etc

Hey,

Out of the patches that are out of tree the ones that are still missing are:
 - microcode loader.

I dug around the "old" implementation of microcode_xen and looked at
some old
Red Hat bugs to get an idea of its pedigree. What I am not sure about, and
I would appreciate some feedback on that, is whether:

 - One should not do a microcode once the CPU has gone in VT mode. That is it
   has some HVM guests running? (or PVH)? Is that some bogus out-of-date
   information that was relevant for the first generation CPUs?


a)Anyhow, barring that I looked at how the baremetal version of the microcode
  update works to see if the hypervisor could trap on the MSR writes and
continue
  on with the update.

  The Intel and AMD Linux drivers seem to follow the same pattern:

  1). Find out what the current microcode version is. That on
   Intel) is via cpuid (0x00000001) and potential RDMSR on MSR_IA32_PLATFORM_ID
   AMD) is via cpuid (0x00000001)

   (both of them read the eax register value)
  2). Apply if neccessary:
   Intel) does an wrmslr on MSR_IA32_UCODE_WRITE (with payload), then wrmslr
       to MSR_IA32_UCODE_REV (with 0), do cpuid (to flush the pipeline + L1) 
and then
       rdmsrl MSR_IA32_UCODE_REV to double check.

   AMD) is via doing rdmsr on MSR_AMD64_PATCH_LEVEL (to check) then follow it
via
       wrmslr to MSR_AMD64_PATCH_LOADER and rdmsr of MSR_AMD64_PATCH_LEVEL.


  Great, except that the "blob" that is provided via these MSR is just
an virtual
  address. No size, nothing. Just ''here it is'', and the CPU
has to figure out the
  size and whether the blob is correct by itself.

  That means implementing this in the hypervisor to do continuation of the
microcode
  loading is not an option. 

b) The microcode_xen driver is not an upstream option either - I don''t
remember the
  details of it, but I do recall Boris Petkov being unhappy about it.

c) Anyhow, thinking about kexec solved I wrote a little tool (see attached) that
sure
  enough allows me to update the microcode. But this is not really an option -
unless
  we add some code in /etc/init.d/xencommons to use this program (with more
logic in it)
  and load the latest microcode.

d) The other option is to use the hypervisor loading logic that Jan developed -
it
  works, but it requires changes in dracut (or mkinitrd) to append all of the
  firmwares (for a specific platform - you can''t mix Intel and AMD) and
add it to
  the stanze. This does it for me:

	cat /lib/firmware/ucode-intel/* > /srv/tftpboot/lab/tst035/microcode.bin

   And then this extra piece of stanze makes it work:

   	KERNEL mboot.c32
   	APPEND xen.gz   ucode=2 --- vmlinuz  --- initramfs.cpio.gz --- microcode.bin


e) A variation of this - is to piggyback on the early-microcode code work done
  by Intel (and AMD), where they construct an cpio image with microcodes and
append it to
  the initrd and scan for a known signature during the boot. The nice thing is
that it
  is generic (can have both AMD and Intel blobs) - Linux does it an Xen can do
it too.
  (See Documentation/x86/early-microcode.txt). Problem is I can''t find
any
  tools (dracut, mkinitrd, etc) that implement it. The tools (dracut) would
probably do:


	mkdir initrd
	cd initrd
	mkdir kernel
	mkdir kernel/x86
	mkdir kernel/x86/microcode
	cp /srv/tftpboot/lab/tst035/microcode.bin kernel/x86/microcode/GenuineIntel.bin
	cp /srv/tftpboot/lab/tst035/amd-microcode.bin
kernel/x86/microcode/AuthenticAMD.bin
	find .|cpio -oc >../ucode.cpio
	cd ..
	cat ucode.cpio /boot/initrd-3.5.0.img >/boot/initrd-3.5.0.ucode.img

  (lifted from said file).

  Anyhow, the neat thing about e) is that once the tools have this, we can just
piggyback
  on it by scanning for the signature and then be able to load the microcode.


I am leaning towards e) b/c it would allow us to:
 - automatically during bootup find the microcode
 - one extra blob for AMD and Intel platforms.
 - generic - as Linux OS can use it as well.

Thoughts?


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On 07/03/2013 04:17 PM, Konrad Rzeszutek Wilk wrote:
> Hey,
>
> Out of the patches that are out of tree the ones that are still missing
are:
>   - microcode loader.
>
> I dug around the "old" implementation of microcode_xen and looked
at some old
> Red Hat bugs to get an idea of its pedigree. What I am not sure about, and
> I would appreciate some feedback on that, is whether:
>
>   - One should not do a microcode once the CPU has gone in VT mode. That is
it
>     has some HVM guests running? (or PVH)? Is that some bogus out-of-date
>     information that was relevant for the first generation CPUs?
>
>
> a)Anyhow, barring that I looked at how the baremetal version of the
microcode
>    update works to see if the hypervisor could trap on the MSR writes and
continue
>    on with the update.
>
>    The Intel and AMD Linux drivers seem to follow the same pattern:
>
>    1). Find out what the current microcode version is. That on
>     Intel) is via cpuid (0x00000001) and potential RDMSR on
MSR_IA32_PLATFORM_ID
>     AMD) is via cpuid (0x00000001)
>
>     (both of them read the eax register value)
>    2). Apply if neccessary:
>     Intel) does an wrmslr on MSR_IA32_UCODE_WRITE (with payload), then
wrmslr
>         to MSR_IA32_UCODE_REV (with 0), do cpuid (to flush the pipeline +
L1)  and then
>         rdmsrl MSR_IA32_UCODE_REV to double check.
>
>     AMD) is via doing rdmsr on MSR_AMD64_PATCH_LEVEL (to check) then follow
it via
>         wrmslr to MSR_AMD64_PATCH_LOADER and rdmsr of
MSR_AMD64_PATCH_LEVEL.
>
>
>    Great, except that the "blob" that is provided via these MSR
is just an virtual
>    address. No size, nothing. Just ''here it is'', and the
CPU has to figure out the
>    size and whether the blob is correct by itself.
HW usually verifies the blob (at least on AMD processors) so if we are 
willing to trust dom0 for sanity checking the blob then intercepting MSR 
write could be an option. Of course, this will require new code path in 
the hypervisor.

However, I think it would be preferable to load microcode earlier than 
during dom0 boot so (d) or (e) would be a better approach.
>
>    That means implementing this in the hypervisor to do continuation of the
microcode
>    loading is not an option.
>
> b) The microcode_xen driver is not an upstream option either - I
don''t remember the
>    details of it, but I do recall Boris Petkov being unhappy about it.
>
> c) Anyhow, thinking about kexec solved I wrote a little tool (see attached)
that sure
>    enough allows me to update the microcode. But this is not really an
option - unless
>    we add some code in /etc/init.d/xencommons to use this program (with
more logic in it)
>    and load the latest microcode.
>
> d) The other option is to use the hypervisor loading logic that Jan
developed - it
>    works, but it requires changes in dracut (or mkinitrd) to append all of
the
>    firmwares (for a specific platform - you can''t mix Intel and
AMD) and add it to
>    the stanze. This does it for me:
>
> 	cat /lib/firmware/ucode-intel/* >
/srv/tftpboot/lab/tst035/microcode.bin
>
>     And then this extra piece of stanze makes it work:
>
>     	KERNEL mboot.c32
>     	APPEND xen.gz   ucode=2 --- vmlinuz  --- initramfs.cpio.gz ---
microcode.bin
>
>
> e) A variation of this - is to piggyback on the early-microcode code work
done
>    by Intel (and AMD), where they construct an cpio image with microcodes
and append it to
>    the initrd and scan for a known signature during the boot. The nice
thing is that it
>    is generic (can have both AMD and Intel blobs) - Linux does it an Xen
can do it too.
>    (See Documentation/x86/early-microcode.txt). Problem is I can''t
find any
>    tools (dracut, mkinitrd, etc) that implement it. The tools (dracut)
would probably do:
>
>
> 	mkdir initrd
> 	cd initrd
> 	mkdir kernel
> 	mkdir kernel/x86
> 	mkdir kernel/x86/microcode
> 	cp /srv/tftpboot/lab/tst035/microcode.bin
kernel/x86/microcode/GenuineIntel.bin
> 	cp /srv/tftpboot/lab/tst035/amd-microcode.bin
kernel/x86/microcode/AuthenticAMD.bin
> 	find .|cpio -oc >../ucode.cpio
> 	cd ..
> 	cat ucode.cpio /boot/initrd-3.5.0.img >/boot/initrd-3.5.0.ucode.img
>
>    (lifted from said file).
>
>    Anyhow, the neat thing about e) is that once the tools have this, we can
just piggyback
>    on it by scanning for the signature and then be able to load the
microcode.
>
>
> I am leaning towards e) b/c it would allow us to:
>   - automatically during bootup find the microcode
>   - one extra blob for AMD and Intel platforms.
>   - generic - as Linux OS can use it as well.
>
> Thoughts?

We probably need both options --- load microcode during boot and after 
system is already up. So your code in (c) is still useful.

-boris

>>> On 03.07.13 at 23:43, Boris Ostrovsky
<boris.ostrovsky@oracle.com> wrote:
> However, I think it would be preferable to load microcode earlier than 
> during dom0 boot so (d) or (e) would be a better approach.
> [...]
> We probably need both options --- load microcode during boot and after 
> system is already up. So your code in (c) is still useful.
Fully agree on both points. As to (d) vs (e) - I personally favor (d)
as not making Xen dependent on Linux details (initrd contents), but
certainly don''t mind (e) getting supported as an alternative (as
long as the amount of code needed for peeking into the initrd isn''t
too huge, and as long as this peeking can be done in a reliable
way, i.e. no false positives).

Jan

On 03/07/13 21:17, Konrad Rzeszutek Wilk wrote:
> Hey,
>
> Out of the patches that are out of tree the ones that are still missing
are:
>  - microcode loader.
>
> I dug around the "old" implementation of microcode_xen and looked
at some old
> Red Hat bugs to get an idea of its pedigree. What I am not sure about, and
> I would appreciate some feedback on that, is whether:
>
>  - One should not do a microcode once the CPU has gone in VT mode. That is
it
>    has some HVM guests running? (or PVH)? Is that some bogus out-of-date
>    information that was relevant for the first generation CPUs?
>
>
> a)Anyhow, barring that I looked at how the baremetal version of the
microcode
>   update works to see if the hypervisor could trap on the MSR writes and
continue
>   on with the update.
>
>   The Intel and AMD Linux drivers seem to follow the same pattern:
>
>   1). Find out what the current microcode version is. That on
>    Intel) is via cpuid (0x00000001) and potential RDMSR on
MSR_IA32_PLATFORM_ID
>    AMD) is via cpuid (0x00000001)
>
>    (both of them read the eax register value)
>   2). Apply if neccessary:
>    Intel) does an wrmslr on MSR_IA32_UCODE_WRITE (with payload), then
wrmslr
>        to MSR_IA32_UCODE_REV (with 0), do cpuid (to flush the pipeline +
L1)  and then
>        rdmsrl MSR_IA32_UCODE_REV to double check.
>
>    AMD) is via doing rdmsr on MSR_AMD64_PATCH_LEVEL (to check) then follow
it via
>        wrmslr to MSR_AMD64_PATCH_LOADER and rdmsr of MSR_AMD64_PATCH_LEVEL.
>
>
>   Great, except that the "blob" that is provided via these MSR is
just an virtual
>   address. No size, nothing. Just ''here it is'', and the
CPU has to figure out the
>   size and whether the blob is correct by itself.
>
>   That means implementing this in the hypervisor to do continuation of the
microcode
>   loading is not an option. 
This is just something we are going to have to accept and work with.  I
hope that loading microcode would take orders of milliseconds, but
perhaps there should be consideration about quiescing normal activity.
>
> b) The microcode_xen driver is not an upstream option either - I
don''t remember the
>   details of it, but I do recall Boris Petkov being unhappy about it.
>
> c) Anyhow, thinking about kexec solved I wrote a little tool (see attached)
that sure
>   enough allows me to update the microcode. But this is not really an
option - unless
>   we add some code in /etc/init.d/xencommons to use this program (with more
logic in it)
>   and load the latest microcode.
>
> d) The other option is to use the hypervisor loading logic that Jan
developed - it
>   works, but it requires changes in dracut (or mkinitrd) to append all of
the
>   firmwares (for a specific platform - you can''t mix Intel and
AMD) and add it to
>   the stanze. This does it for me:
>
> 	cat /lib/firmware/ucode-intel/* >
/srv/tftpboot/lab/tst035/microcode.bin
>
>    And then this extra piece of stanze makes it work:
>
>    	KERNEL mboot.c32
>    	APPEND xen.gz   ucode=2 --- vmlinuz  --- initramfs.cpio.gz ---
microcode.bin
>
>
> e) A variation of this - is to piggyback on the early-microcode code work
done
>   by Intel (and AMD), where they construct an cpio image with microcodes
and append it to
>   the initrd and scan for a known signature during the boot. The nice thing
is that it
>   is generic (can have both AMD and Intel blobs) - Linux does it an Xen can
do it too.
>   (See Documentation/x86/early-microcode.txt). Problem is I can''t
find any
>   tools (dracut, mkinitrd, etc) that implement it. The tools (dracut) would
probably do:
>
>
> 	mkdir initrd
> 	cd initrd
> 	mkdir kernel
> 	mkdir kernel/x86
> 	mkdir kernel/x86/microcode
> 	cp /srv/tftpboot/lab/tst035/microcode.bin
kernel/x86/microcode/GenuineIntel.bin
> 	cp /srv/tftpboot/lab/tst035/amd-microcode.bin
kernel/x86/microcode/AuthenticAMD.bin
> 	find .|cpio -oc >../ucode.cpio
> 	cd ..
> 	cat ucode.cpio /boot/initrd-3.5.0.img >/boot/initrd-3.5.0.ucode.img
>
>   (lifted from said file).
>
>   Anyhow, the neat thing about e) is that once the tools have this, we can
just piggyback
>   on it by scanning for the signature and then be able to load the
microcode.
>
>
> I am leaning towards e) b/c it would allow us to:
>  - automatically during bootup find the microcode
>  - one extra blob for AMD and Intel platforms.
>  - generic - as Linux OS can use it as well.
>
> Thoughts?
Getting the microcode loaded earlier is likely to be better, so d) would
be preferable for that.

~Andrew
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel