Linux Virtualization - Sep 2014 - Standardizing an MSR or other hypercall to get an RNG seed?

On Fri, Sep 19, 2014 at 11:02:38AM -0700, Andy Lutomirski
wrote:
> On Fri, Sep 19, 2014 at 10:49 AM, Gleb Natapov <gleb at kernel.org>
wrote:
> > On Fri, Sep 19, 2014 at 10:18:37AM -0700, H. Peter Anvin wrote:
> >> On 09/19/2014 10:15 AM, Gleb Natapov wrote:
> >> > On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter Anvin
wrote:
> >> >> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
> >> >>> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H. Peter
Anvin wrote:
> >> >>>> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
> >> >>>>>
> >> >>>>> Linux detects what hypervior it runs on very
early
> >> >>>>
> >> >>>> Not anywhere close to early enough.  We're
talking for uses like kASLR.
> >> >>>>
> >> >>> Still to early to do:
> >> >>>
> >> >>>    h = cpuid(HYPERVIOR_SIGNATURE)
> >> >>>    if (h == KVMKVMKVM) {
> >> >>>       if (cpuid(kvm_features) & kvm_rnd)
> >> >>>          rdmsr(kvm_rnd)
> >> >>>    else (h == HyperV) {
> >> >>>       if (cpuid(hv_features) & hv_rnd)
> >> >>>         rdmsr(hv_rnd)
> >> >>>    else (h == XenXenXen) {
> >> >>>       if (cpuid(xen_features) & xen_rnd)
> >> >>>         rdmsr(xen_rnd)
> >> >>>   }
> >> >>>
> >> >>
> >> >> If we need to do chase loops, especially not so...
> >> >>
> >> > What loops exactly? As a non native English speaker I fail to
understand
> >> > if your answer is "yes" or "no" ;)
> >> >
> >>
> >> The above isn't actually the full algorithm used.
> >>
> > What part of actually algorithm cannot be implemented? Loop that
searches
> > for KVM leaf in case KVM pretend to be HyperV (is this what you called
> > "chase loops"?)? First of all there is no need to implement
it, if KVM
> > pretends to be HyperV use HyperV's way to obtain RNG, but what is
the
> > problem with the loop?
> >
> 
> It can be implemented, and I've done it.  But it's a mess.  Almost
the
> very first thing we do in boot (even before decompressing the kernel)
> will be to scan a bunch of cpuid leaves looking for a hypervisor with
> an rng source that we can use for kASLR.  And we'll have to update
> that code and make it bigger every time another hypervisor adds
> exactly the same feature.
IMO implementing this feature is in hypervisor's best interest, so the task
of updating the code will scale by virtue of hypervisor's developers each
adding it for hypervisor he cares about.
> 
> And then we have another copy of almost exactly the same code in the
> normal post-boot part of the kernel.
> 
> We can certainly do this, but I'd much rather solve the problem once
> and let all of the hypervisors and guests opt in and immediately be
> compatible with each other.
> 
> > I "forgot" VMware because I do not see VMware people to be
CCed. They may
> > be even less excited about them being told _how_ this feature need to
be
> > implemented (e.g implement HyperV leafs for the feature detection). I
> > do not want to and cannot speak for VMware, but my guess is that for
> > them it would be much easier to add an else clause for VMware in above
> > "if" then to coordinate with all hypervisor developers about
MSR/cpuid
> > details. And since this is security feature implementing it for Linux
> > is in their best interest.
> 
> Do you know any of them who should be cc'd?
> 
No, not anyone in particular. git log arch/x86/kernel/cpu/vmware.c may help.

But VMware is an elephant in the room here. There are other hypervisors out
there.
VirtualBox, bhyve...

--
			Gleb.

[cc: Alok Kataria at VMware]

On Fri, Sep 19, 2014 at 11:12 AM, Gleb Natapov <gleb at kernel.org>
wrote:
> On Fri, Sep 19, 2014 at 11:02:38AM -0700, Andy Lutomirski wrote:
>> On Fri, Sep 19, 2014 at 10:49 AM, Gleb Natapov <gleb at
kernel.org> wrote:
>> > On Fri, Sep 19, 2014 at 10:18:37AM -0700, H. Peter Anvin wrote:
>> >> On 09/19/2014 10:15 AM, Gleb Natapov wrote:
>> >> > On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter Anvin
wrote:
>> >> >> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
>> >> >>> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H.
Peter Anvin wrote:
>> >> >>>> On 09/19/2014 09:37 AM, Gleb Natapov wrote:
>> >> >>>>>
>> >> >>>>> Linux detects what hypervior it runs on
very early
>> >> >>>>
>> >> >>>> Not anywhere close to early enough. 
We're talking for uses like kASLR.
>> >> >>>>
>> >> >>> Still to early to do:
>> >> >>>
>> >> >>>    h = cpuid(HYPERVIOR_SIGNATURE)
>> >> >>>    if (h == KVMKVMKVM) {
>> >> >>>       if (cpuid(kvm_features) & kvm_rnd)
>> >> >>>          rdmsr(kvm_rnd)
>> >> >>>    else (h == HyperV) {
>> >> >>>       if (cpuid(hv_features) & hv_rnd)
>> >> >>>         rdmsr(hv_rnd)
>> >> >>>    else (h == XenXenXen) {
>> >> >>>       if (cpuid(xen_features) & xen_rnd)
>> >> >>>         rdmsr(xen_rnd)
>> >> >>>   }
>> >> >>>
>> >> >>
>> >> >> If we need to do chase loops, especially not so...
>> >> >>
>> >> > What loops exactly? As a non native English speaker I
fail to understand
>> >> > if your answer is "yes" or "no" ;)
>> >> >
>> >>
>> >> The above isn't actually the full algorithm used.
>> >>
>> > What part of actually algorithm cannot be implemented? Loop that
searches
>> > for KVM leaf in case KVM pretend to be HyperV (is this what you
called
>> > "chase loops"?)? First of all there is no need to
implement it, if KVM
>> > pretends to be HyperV use HyperV's way to obtain RNG, but what
is the
>> > problem with the loop?
>> >
>>
>> It can be implemented, and I've done it.  But it's a mess. 
Almost the
>> very first thing we do in boot (even before decompressing the kernel)
>> will be to scan a bunch of cpuid leaves looking for a hypervisor with
>> an rng source that we can use for kASLR.  And we'll have to update
>> that code and make it bigger every time another hypervisor adds
>> exactly the same feature.
> IMO implementing this feature is in hypervisor's best interest, so the
task
> of updating the code will scale by virtue of hypervisor's developers
each
> adding it for hypervisor he cares about.
I assume that you mean guest, not hypervisor.
>
>>
>> And then we have another copy of almost exactly the same code in the
>> normal post-boot part of the kernel.
>>
>> We can certainly do this, but I'd much rather solve the problem
once
>> and let all of the hypervisors and guests opt in and immediately be
>> compatible with each other.
>>
>> > I "forgot" VMware because I do not see VMware people to
be CCed. They may
>> > be even less excited about them being told _how_ this feature need
to be
>> > implemented (e.g implement HyperV leafs for the feature
detection). I
>> > do not want to and cannot speak for VMware, but my guess is that
for
>> > them it would be much easier to add an else clause for VMware in
above
>> > "if" then to coordinate with all hypervisor developers
about MSR/cpuid
>> > details. And since this is security feature implementing it for
Linux
>> > is in their best interest.
>>
>> Do you know any of them who should be cc'd?
>>
> No, not anyone in particular. git log arch/x86/kernel/cpu/vmware.c may
help.
>
> But VMware is an elephant in the room here. There are other hypervisors out
there.
> VirtualBox, bhyve...
Exactly.  The amount of effort to get everything to be compatible with
everything scales quadratically in the number of hypervisors, and the
probability that some combination is broken also increases.

If we can get everyone to back something common here then this problem
goes away.

--Andy
>
> --
>                         Gleb.


-- 
Andy Lutomirski
AMA Capital Management, LLC

On Fri, Sep 19, 2014 at 11:20:49AM -0700, Andy Lutomirski
wrote:
> [cc: Alok Kataria at VMware]
> 
> On Fri, Sep 19, 2014 at 11:12 AM, Gleb Natapov <gleb at kernel.org>
wrote:
> > On Fri, Sep 19, 2014 at 11:02:38AM -0700, Andy Lutomirski wrote:
> >> On Fri, Sep 19, 2014 at 10:49 AM, Gleb Natapov <gleb at
kernel.org> wrote:
> >> > On Fri, Sep 19, 2014 at 10:18:37AM -0700, H. Peter Anvin
wrote:
> >> >> On 09/19/2014 10:15 AM, Gleb Natapov wrote:
> >> >> > On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter
Anvin wrote:
> >> >> >> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
> >> >> >>> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H.
Peter Anvin wrote:
> >> >> >>>> On 09/19/2014 09:37 AM, Gleb Natapov
wrote:
> >> >> >>>>>
> >> >> >>>>> Linux detects what hypervior it runs
on very early
> >> >> >>>>
> >> >> >>>> Not anywhere close to early enough. 
We're talking for uses like kASLR.
> >> >> >>>>
> >> >> >>> Still to early to do:
> >> >> >>>
> >> >> >>>    h = cpuid(HYPERVIOR_SIGNATURE)
> >> >> >>>    if (h == KVMKVMKVM) {
> >> >> >>>       if (cpuid(kvm_features) & kvm_rnd)
> >> >> >>>          rdmsr(kvm_rnd)
> >> >> >>>    else (h == HyperV) {
> >> >> >>>       if (cpuid(hv_features) & hv_rnd)
> >> >> >>>         rdmsr(hv_rnd)
> >> >> >>>    else (h == XenXenXen) {
> >> >> >>>       if (cpuid(xen_features) & xen_rnd)
> >> >> >>>         rdmsr(xen_rnd)
> >> >> >>>   }
> >> >> >>>
> >> >> >>
> >> >> >> If we need to do chase loops, especially not
so...
> >> >> >>
> >> >> > What loops exactly? As a non native English speaker
I fail to understand
> >> >> > if your answer is "yes" or "no"
;)
> >> >> >
> >> >>
> >> >> The above isn't actually the full algorithm used.
> >> >>
> >> > What part of actually algorithm cannot be implemented? Loop
that searches
> >> > for KVM leaf in case KVM pretend to be HyperV (is this what
you called
> >> > "chase loops"?)? First of all there is no need to
implement it, if KVM
> >> > pretends to be HyperV use HyperV's way to obtain RNG, but
what is the
> >> > problem with the loop?
> >> >
> >>
> >> It can be implemented, and I've done it.  But it's a mess.
Almost the
> >> very first thing we do in boot (even before decompressing the
kernel)
> >> will be to scan a bunch of cpuid leaves looking for a hypervisor
with
> >> an rng source that we can use for kASLR.  And we'll have to
update
> >> that code and make it bigger every time another hypervisor adds
> >> exactly the same feature.
> > IMO implementing this feature is in hypervisor's best interest, so
the task
> > of updating the code will scale by virtue of hypervisor's
developers each
> > adding it for hypervisor he cares about.
> 
> I assume that you mean guest, not hypervisor.
> 
Yes, I mean guest support for hypervisor he cares about.
> >
> >>
> >> And then we have another copy of almost exactly the same code in
the
> >> normal post-boot part of the kernel.
> >>
> >> We can certainly do this, but I'd much rather solve the
problem once
> >> and let all of the hypervisors and guests opt in and immediately
be
> >> compatible with each other.
> >>
> >> > I "forgot" VMware because I do not see VMware
people to be CCed. They may
> >> > be even less excited about them being told _how_ this feature
need to be
> >> > implemented (e.g implement HyperV leafs for the feature
detection). I
> >> > do not want to and cannot speak for VMware, but my guess is
that for
> >> > them it would be much easier to add an else clause for VMware
in above
> >> > "if" then to coordinate with all hypervisor
developers about MSR/cpuid
> >> > details. And since this is security feature implementing it
for Linux
> >> > is in their best interest.
> >>
> >> Do you know any of them who should be cc'd?
> >>
> > No, not anyone in particular. git log arch/x86/kernel/cpu/vmware.c may
help.
> >
> > But VMware is an elephant in the room here. There are other
hypervisors out there.
> > VirtualBox, bhyve...
> 
> Exactly.  The amount of effort to get everything to be compatible with
> everything scales quadratically in the number of hypervisors, and the
> probability that some combination is broken also increases.
> 
The effort is distributed equally among hypervisor developers. If they
want Linux to be more secure on their hypervisor they contribute guest
code. They do need to write hypervisor part anyway. On cpus with RDRAND
instruction this MSR is not even needed and some hypervisors may decide
that support for old cpus does not worth the effort. Unified interface
does not help if hypervisor does not implement it.

--
			Gleb.

Hi Andy,

On Fri, 2014-09-19 at 11:20 -0700, Andy Lutomirski
wrote:
> [cc: Alok Kataria at VMware]
> 
> On Fri, Sep 19, 2014 at 11:12 AM, Gleb Natapov <gleb at kernel.org>
wrote:
> > On Fri, Sep 19, 2014 at 11:02:38AM -0700, Andy Lutomirski wrote:
> >> On Fri, Sep 19, 2014 at 10:49 AM, Gleb Natapov <gleb at
kernel.org> wrote:
> >> > On Fri, Sep 19, 2014 at 10:18:37AM -0700, H. Peter Anvin
wrote:
> >> >> On 09/19/2014 10:15 AM, Gleb Natapov wrote:
> >> >> > On Fri, Sep 19, 2014 at 10:08:20AM -0700, H. Peter
Anvin wrote:
> >> >> >> On 09/19/2014 09:53 AM, Gleb Natapov wrote:
> >> >> >>> On Fri, Sep 19, 2014 at 09:40:07AM -0700, H.
Peter Anvin wrote:
> >> >> >>>> On 09/19/2014 09:37 AM, Gleb Natapov
wrote:
> >> >> >>>>>
> >> >> >>>>> Linux detects what hypervior it runs
on very early
> >> >> >>>>
> >> >> >>>> Not anywhere close to early enough. 
We're talking for uses like kASLR.
> >> >> >>>>
> >> >> >>> Still to early to do:
> >> >> >>>
> >> >> >>>    h = cpuid(HYPERVIOR_SIGNATURE)
> >> >> >>>    if (h == KVMKVMKVM) {
> >> >> >>>       if (cpuid(kvm_features) & kvm_rnd)
> >> >> >>>          rdmsr(kvm_rnd)
> >> >> >>>    else (h == HyperV) {
> >> >> >>>       if (cpuid(hv_features) & hv_rnd)
> >> >> >>>         rdmsr(hv_rnd)
> >> >> >>>    else (h == XenXenXen) {
> >> >> >>>       if (cpuid(xen_features) & xen_rnd)
> >> >> >>>         rdmsr(xen_rnd)
> >> >> >>>   }
> >> >> >>>
> >> >> >>
> >> >> >> If we need to do chase loops, especially not
so...
> >> >> >>
> >> >> > What loops exactly? As a non native English speaker
I fail to understand
> >> >> > if your answer is "yes" or "no"
;)
> >> >> >
> >> >>
> >> >> The above isn't actually the full algorithm used.
> >> >>
> >> > What part of actually algorithm cannot be implemented? Loop
that searches
> >> > for KVM leaf in case KVM pretend to be HyperV (is this what
you called
> >> > "chase loops"?)? First of all there is no need to
implement it, if KVM
> >> > pretends to be HyperV use HyperV's way to obtain RNG, but
what is the
> >> > problem with the loop?
> >> >
> >>
> >> It can be implemented, and I've done it.  But it's a mess.
Almost the
> >> very first thing we do in boot (even before decompressing the
kernel)
> >> will be to scan a bunch of cpuid leaves looking for a hypervisor
with
> >> an rng source that we can use for kASLR.  And we'll have to
update
> >> that code and make it bigger every time another hypervisor adds
> >> exactly the same feature.
> > IMO implementing this feature is in hypervisor's best interest, so
the task
> > of updating the code will scale by virtue of hypervisor's
developers each
> > adding it for hypervisor he cares about.
> 
> I assume that you mean guest, not hypervisor.
> 
> >
> >>
> >> And then we have another copy of almost exactly the same code in
the
> >> normal post-boot part of the kernel.
> >>
> >> We can certainly do this, but I'd much rather solve the
problem once
> >> and let all of the hypervisors and guests opt in and immediately
be
> >> compatible with each other.
> >>
> >> > I "forgot" VMware because I do not see VMware
people to be CCed. They may
> >> > be even less excited about them being told _how_ this feature
need to be
> >> > implemented (e.g implement HyperV leafs for the feature
detection). I
> >> > do not want to and cannot speak for VMware, but my guess is
that for
> >> > them it would be much easier to add an else clause for VMware
in above
> >> > "if" then to coordinate with all hypervisor
developers about MSR/cpuid
> >> > details. And since this is security feature implementing it
for Linux
> >> > is in their best interest.
> >>
> >> Do you know any of them who should be cc'd?
> >>
> > No, not anyone in particular. git log arch/x86/kernel/cpu/vmware.c may
help.
> >
> > But VMware is an elephant in the room here. There are other
hypervisors out there.
> > VirtualBox, bhyve...
> 
> Exactly.  The amount of effort to get everything to be compatible with
> everything scales quadratically in the number of hypervisors, and the
> probability that some combination is broken also increases.
> 
> If we can get everyone to back something common here then this problem
> goes away.
There was a similar attempt few years back [1], to standardize on the
hypervisor cpuid space. Though a few of them were interested, getting
all hypervisor vendors to agree (actually even discuss this) turned out
to be a futile exercise. Don't mean to discourage you, but what I
learned from that attempt was that it's very difficult to standardize
unless the hardware vendors are proposing it.

In anycase can you point me to a mail which discusses the specifics of
the interface you are proposing ? 

Alok

[1] - http://thread.gmane.org/gmane.comp.emulators.kvm.devel/22643
      https://lkml.org/lkml/2008/9/26/351