Alok Kataria
2008-Oct-01 17:14 UTC
[RFC] CPUID usage for interaction between Hypervisors and Linux.
Hi, Please find below the proposal for the generic use of cpuid space allotted for hypervisors. Apart from this cpuid space another thing worth noting would be that, Intel & AMD reserve the MSRs from 0x40000000 - 0x400000FF for software use. Though the proposal doesn't talk about MSR's right now, we should be aware of these reservations as we may want to extend the way we use CPUID to MSR usage as well. While we are at it, we also think we should form a group which has at least one person representing each of the hypervisors interested in generalizing the hypervisor CPUID space for Linux guest OS. This group will be informed whenever a new CPUID leaf from the generic space is to be used. This would help avoid any duplicate definitions for a CPUID semantic by two different hypervisors. I think most of the people are subscribed to LKML or the virtualization lists and we should use these lists as a platform to decide on things. Thanks, Alok --- Hypervisor CPUID Interface Proposal ----------------------------------- Intel & AMD have reserved cpuid levels 0x40000000 - 0x400000FF for software use. Hypervisors can use these levels to provide an interface to pass information from the hypervisor to the guest running inside a virtual machine. This proposal defines a standard framework for the way in which the Linux and hypervisor communities incrementally define this CPUID space. (This proposal may be adopted by other guest OSes. However, that is not a requirement because a hypervisor can expose a different CPUID interface depending on the guest OS type that is specified by the VM configuration.) Hypervisor Present Bit: Bit 31 of ECX of CPUID leaf 0x1. This bit has been reserved by Intel & AMD for use by hypervisors, and indicates the presence of a hypervisor. Virtual CPU's (hypervisors) set this bit to 1 and physical CPU's (all existing and future cpu's) set this bit to zero. This bit can be probed by the guest software to detect whether they are running inside a virtual machine. Hypervisor CPUID Information Leaf: Leaf 0x40000000. This leaf returns the CPUID leaf range supported by the hypervisor and the hypervisor vendor signature. # EAX: The maximum input value for CPUID supported by the hypervisor. # EBX, ECX, EDX: Hypervisor vendor ID signature. Hypervisor Specific Leaves: Leaf range 0x40000001 - 0x4000000F. These cpuid leaves are reserved as hypervisor specific leaves. The semantics of these 15 leaves depend on the signature read from the "Hypervisor Information Leaf". Generic Leaves: Leaf range 0x40000010 - 0x4000000FF. The semantics of these leaves are consistent across all hypervisors. This allows the guest kernel to probe and interpret these leaves without checking for a hypervisor signature. A hypervisor can indicate that a leaf or a leaf's field is unsupported by returning zero when that leaf or field is probed. To avoid the situation where multiple hypervisors attempt to define the semantics for the same leaf during development, we can partition the generic leaf space to allow each hypervisor to define a part of the generic space. For instance: VMware could define 0x4000001X Xen could define 0x4000002X KVM could define 0x4000003X and so on... Note that hypervisors can implement any leaves that have been defined in the generic leaf space whenever common features can be found. For example, VMware hypervisors can implement leafs that have been defined in the KVM area 0x4000003X and vice versa. The kernel can detect the support for a generic field inside leaf 0x400000XY using the following algorithm: 1. Get EAX from Leaf 0x400000000, Hypervisor CPUID information. EAX returns the maximum input value for the hypervisor CPUID space. If EAX < 0x400000XY, then the field is not available. 2. Else, extract the field from the target Leaf 0x400000XY by doing cpuid(0x400000XY). If (field == 0), this feature is unsupported/unimplemented by the hypervisor. The kernel should handle this case gracefully so that a hypervisor is never required to support or implement any particular generic leaf. -------------------------------------------------------------------------------- Definition of the Generic CPUID space. Leaf 0x40000010, Timing Information. VMware has defined the first generic leaf to provide timing information. This leaf returns the current TSC frequency and current Bus frequency in kHz. # EAX: (Virtual) TSC frequency in kHz. # EBX: (Virtual) Bus (local apic timer) frequency in kHz. # ECX, EDX: RESERVED (Per above, reserved fields are set to zero). -------------------------------------------------------------------------------- Written By, Alok N Kataria <akataria at vmware.com> Dan Hecht <dhecht at vmware.com> Inputs from, Jun Nakajima <jun.nakajima at intel.com>
H. Peter Anvin
2008-Oct-01 17:21 UTC
[RFC] CPUID usage for interaction between Hypervisors and Linux.
Alok Kataria wrote:> > (This proposal may be adopted by other guest OSes. However, that is not > a requirement because a hypervisor can expose a different CPUID > interface depending on the guest OS type that is specified by the VM > configuration.) >Excuse me, but that is blatantly idiotic. Expecting the user having to configure a VM to match the target OS is *exactly* as stupid as expecting the user to reconfigure the BIOS. It's totally the wrong thing to do. -hpa
H. Peter Anvin
2008-Oct-01 17:47 UTC
[RFC] CPUID usage for interaction between Hypervisors and Linux.
Alok Kataria wrote:> > Hypervisor CPUID Interface Proposal > ----------------------------------- > > Intel & AMD have reserved cpuid levels 0x40000000 - 0x400000FF for > software use. Hypervisors can use these levels to provide an interface > to pass information from the hypervisor to the guest running inside a > virtual machine. > > This proposal defines a standard framework for the way in which the > Linux and hypervisor communities incrementally define this CPUID space. >I also observe that your proposal provides no mean of positive identification, i.e. that a hypervisor actually conforms to your proposal. -hpa
Jeremy Fitzhardinge
2008-Oct-01 18:04 UTC
[RFC] CPUID usage for interaction between Hypervisors and Linux.
Alok Kataria wrote:> Hi, > > Please find below the proposal for the generic use of cpuid space > allotted for hypervisors. Apart from this cpuid space another thing > worth noting would be that, Intel & AMD reserve the MSRs from 0x40000000 > - 0x400000FF for software use. Though the proposal doesn't talk about > MSR's right now, we should be aware of these reservations as we may want > to extend the way we use CPUID to MSR usage as well. > > While we are at it, we also think we should form a group which has at > least one person representing each of the hypervisors interested in > generalizing the hypervisor CPUID space for Linux guest OS. This group > will be informed whenever a new CPUID leaf from the generic space is to > be used. This would help avoid any duplicate definitions for a CPUID > semantic by two different hypervisors. I think most of the people are > subscribed to LKML or the virtualization lists and we should use these > lists as a platform to decide on things. > > Thanks, > Alok > > --- > > Hypervisor CPUID Interface Proposal > ----------------------------------- > > Intel & AMD have reserved cpuid levels 0x40000000 - 0x400000FF for > software use. Hypervisors can use these levels to provide an interface > to pass information from the hypervisor to the guest running inside a > virtual machine. > > This proposal defines a standard framework for the way in which the > Linux and hypervisor communities incrementally define this CPUID space. > > (This proposal may be adopted by other guest OSes. However, that is not > a requirement because a hypervisor can expose a different CPUID > interface depending on the guest OS type that is specified by the VM > configuration.) > > Hypervisor Present Bit: > Bit 31 of ECX of CPUID leaf 0x1. > > This bit has been reserved by Intel & AMD for use by > hypervisors, and indicates the presence of a hypervisor. > > Virtual CPU's (hypervisors) set this bit to 1 and physical CPU's > (all existing and future cpu's) set this bit to zero. This bit > can be probed by the guest software to detect whether they are > running inside a virtual machine. > > Hypervisor CPUID Information Leaf: > Leaf 0x40000000. > > This leaf returns the CPUID leaf range supported by the > hypervisor and the hypervisor vendor signature. > > # EAX: The maximum input value for CPUID supported by the hypervisor. > # EBX, ECX, EDX: Hypervisor vendor ID signature. > > Hypervisor Specific Leaves: > Leaf range 0x40000001 - 0x4000000F. > > These cpuid leaves are reserved as hypervisor specific leaves. > The semantics of these 15 leaves depend on the signature read > from the "Hypervisor Information Leaf". > > Generic Leaves: > Leaf range 0x40000010 - 0x4000000FF. > > The semantics of these leaves are consistent across all > hypervisors. This allows the guest kernel to probe and > interpret these leaves without checking for a hypervisor > signature. > > A hypervisor can indicate that a leaf or a leaf's field is > unsupported by returning zero when that leaf or field is probed. > > To avoid the situation where multiple hypervisors attempt to define the > semantics for the same leaf during development, we can partition > the generic leaf space to allow each hypervisor to define a part > of the generic space. > > For instance: > VMware could define 0x4000001X > Xen could define 0x4000002X > KVM could define 0x4000003X > and so on... >No, we're not getting anywhere. This is an outright broken idea. The space is too small to be able to chop up in this way, and the number of vendors too large to be able to do it without having a central oversight. The only way this can work is by having explicit positive identification of each group of leaves with a signature. If there's a recognizable signature, then you can inspect the rest of the group; if not, then you can't. That way, you can avoid any leaf usage which doesn't conform to this model, and you can also simultaneously support multiple hypervisor ABIs. It also accommodates existing hypervisor use of this leaf space, even if they currently use a fixed location within it. A concrete counter-proposal: The space 0x40000000-0x400000ff is reserved for hypervisor usage. This region is divided into 16 16-leaf blocks. Each block has the structure: 0x400000x0: eax: max used leaf within the leaf block (max 0x400000xf) e[bcd]x: leaf block signature. This may be a hypervisor-specific signature, or a generic signature, depending on the contents of the block A guest may search for any supported Hypervisor ABIs by inspecting each leaf at 0x400000x0 for a known signature, and then may choose its mode of operation accordingly. It must ignore any unknown signatures, and not touch any of the leaves within an unknown leaf block. Hypervisor vendors who want to add a hypervisor-specific leaf block must choose a signature which is recognizably related to their or their hypervisor's name. Signatures starting with "Generic" are reserved for generic leaf blocks. A guest may scan leaf blocks to enumerate what hypervisor ABIs/hypercall interfaces are available to it. It may mix and match any information from leaves it understands. However, once it starts using a specific hypervisor ABI by making hypercalls or doing other operations with side-effects, it must commit to using that ABI exclusively (a specific hypervisor ABI may include the generic ABI by reference, however). Correspondingly, a hypervisor must treat any cpuid accesses as side-effect free. Definition of specific blocks: Generic hypervisor leaf block: 0x400000x0 signature is "GenericVMMIF" (or something) 0x400000x1 tsc leaf as you've described J
Anthony Liguori
2008-Oct-01 20:03 UTC
[RFC] CPUID usage for interaction between Hypervisors and Linux.
Jeremy Fitzhardinge wrote:> Alok Kataria wrote: > > No, we're not getting anywhere. This is an outright broken idea. The > space is too small to be able to chop up in this way, and the number of > vendors too large to be able to do it without having a central oversight. > > The only way this can work is by having explicit positive identification > of each group of leaves with a signature. If there's a recognizable > signature, then you can inspect the rest of the group; if not, then you > can't. That way, you can avoid any leaf usage which doesn't conform to > this model, and you can also simultaneously support multiple hypervisor > ABIs. It also accommodates existing hypervisor use of this leaf space, > even if they currently use a fixed location within it. > > A concrete counter-proposal:Mmm, cpuid bikeshedding :-)> The space 0x40000000-0x400000ff is reserved for hypervisor usage. > > This region is divided into 16 16-leaf blocks. Each block has the > structure: > > 0x400000x0: > eax: max used leaf within the leaf block (max 0x400000xf)Why even bother with this? It doesn't seem necessary in your proposal. Regards, Anthony Liguori
Anthony Liguori
2008-Oct-01 21:03 UTC
[RFC] CPUID usage for interaction between Hypervisors and Linux.
Jeremy Fitzhardinge wrote:> Anthony Liguori wrote: >> Mmm, cpuid bikeshedding :-) > > My shade of blue is better. > >>> The space 0x40000000-0x400000ff is reserved for hypervisor usage. >>> >>> This region is divided into 16 16-leaf blocks. Each block has the >>> structure: >>> >>> 0x400000x0: >>> eax: max used leaf within the leaf block (max 0x400000xf) >> Why even bother with this? It doesn't seem necessary in your proposal. > > It allows someone to incrementally add things to their block in a fairly > orderly way. But more importantly, its the prevailing idiom, and the > existing and proposed cpuid schemes already do this, so they'd fit in as-is.We just leave eax as zero. It wouldn't be that upsetting to change this as it would only keep new guests from working on older KVMs. However, I see little incentive to change anything unless there's something compelling that we would get in return. Since we're only talking about Linux guests, it's just as easy for us to add things to our paravirt_ops implementation as it would be to add things using this new model. If this was something that other guests were all agreeing to support (even if it was just the BSDs and OpenSolaris), then there may be value to it. Right now, I see no real value in changing the status quo. Regards, Anthony Liguori> J
Possibly Parallel Threads
- [RFC] CPUID usage for interaction between Hypervisors and Linux.
- Standardizing an MSR or other hypercall to get an RNG seed?
- Standardizing an MSR or other hypercall to get an RNG seed?
- [PATCH v2 00/11] x86: xen cpuid() cleanup
- [PATCH v2 00/11] x86: xen cpuid() cleanup