[RFC, PATCH 1/24] i386 Vmi documentation

Zachary Amsden
Wed, 18 Apr 2007 17:49:24 -0000

Index: linux-2.6.16-rc5/Documentation/vmi_spec.txt
==================================================================---
linux-2.6.16-rc5.orig/Documentation/vmi_spec.txt	2006-03-09 23:33:29.000000000
-0800
+++ linux-2.6.16-rc5/Documentation/vmi_spec.txt	2006-03-10 12:55:29.000000000
-0800
@@ -0,0 +1,2197 @@
+
+	Paravirtualization API Version 2.0
+
+	Zachary Amsden, Daniel Arai, Daniel Hecht, Pratap Subrahmanyam
+	Copyright (C) 2005, 2006, VMware, Inc.
+	All rights reserved
+
+Revision history:
+         1.0: Initial version
+         1.1: arai 2005-11-15
+              Added SMP-related sections: AP startup and Local APIC support
+         1.2: dhecht 2006-02-23
+              Added Time Interface section and Time related VMI calls
+
+Contents
+
+1) Motivations
+2) Overview
+    Initialization
+    Privilege model
+    Memory management
+    Segmentation
+    Interrupt and I/O subsystem
+    IDT management
+    Transparent Paravirtualization
+    3rd Party Extensions
+    AP Startup
+    State Synchronization in SMP systems
+    Local APIC Support
+    Time Interface
+3) Architectural Differences from Native Hardware
+4) ROM Implementation
+    Detection
+    Data layout
+    Call convention
+    PCI implementation
+
+Appendix A - VMI ROM low level ABI
+Appendix B - VMI C prototypes
+Appendix C - Sensitive x86 instructions
+
+
+1) Motivations
+
+   There are several high level goals which must be balanced in designing
+   an API for paravirtualization.  The most general concerns are:
+
+   Portability      - it should be easy to port a guest OS to use the API
+   High performance - the API must not obstruct a high performance
+	              hypervisor implementation
+   Maintainability  - it should be easy to maintain and upgrade the guest
+                      OS
+   Extensibility    - it should be possible for future expansion of the
+                      API
+
+   Portability.
+
+     The general approach to paravirtualization rather than full
+     virtualization is to modify the guest operating system.  This means
+     there is implicitly some code cost to port a guest OS to run in a
+     paravirtual environment.  The closer the API resembles a native
+     platform which the OS supports, the lower the cost of porting.
+     Rather than provide an alternative, high level interface for this
+     API, the approach is to provide a low level interface which
+     encapsulates the sensitive and performance critical parts of the
+     system.  Thus, we have direct parallels to most privileged
+     instructions, and the process of converting a guest OS to use these
+     instructions is in many cases a simple replacement of one function
+     for another. Although this is sufficient for CPU virtualization,
+     performance concerns have forced us to add additional calls for
+     memory management, and notifications about updates to certain CPU
+     data structures. Support for this in the Linux operating system has
+     proved to be very minimal in cost because of the already somewhat
+     portable and modular design of the memory management layer.
+
+  High Performance.
+
+     Providing a low level API that closely resembles hardware does not
+     provide any support for compound operations; indeed, typical
+     compound operations on hardware can be updating of many page table
+     entries, flushing system TLBs, or providing floating point safety.
+     Since these operations may require several privileged or sensitive
+     operations, it becomes important to defer some of these operations
+     until explicit flushes are issued, or to provide higher level
+     operations around some of these functions.  In order to keep with
+     the goal of portability, this has been done only when deemed
+     necessary for performance reasons, and we have tried to package
+     these compound operations into methods that are typically used in
+     guest operating systems.  In the future, we envision that additional
+     higher level abstractions will be added as an adjunct to the
+     low-level API.  These higher level abstractions will target large
+     bulk operations such as creation, and destruction of address spaces,
+     context switches, thread creation and control.
+
+  Maintainability.
+
+     In the course of development with a virtualized environment, it is
+     not uncommon for support of new features or higher performance to
+     require radical changes to the operation of the system.  If these
+     changes are visible to the guest OS in a paravirtualized system,
+     this will require updates to the guest kernel, which presents a
+     maintenance problem.  In the Linux world, the rapid pace of
+     development on the kernel means new kernel versions are produced
+     every few months.  This rapid pace is not always appropriate for end
+     users, so it is not uncommon to have dozens of different versions of
+     the Linux kernel in use that must be actively supported.  To keep
+     this many versions in sync with potentially radical changes in the
+     paravirtualized system is not a scalable solution.  To reduce the
+     maintenance burden as much as possible, while still allowing the
+     implementation to accommodate changes, the design provides a stable
+     ABI with semantic invariants.  The underlying implementation of the
+     ABI and details of what data or how it communicates with the
+     hypervisor are not visible to the guest OS.  As a result, in most
+     cases, the guest OS need not even be recompiled to work with a newer
+     hypervisor.  This allows performance optimizations, bug fixes,
+     debugging, or statistical instrumentation to be added to the API
+     implementation without any impact on the guest kernel.  This is
+     achieved by publishing a block of code from the hypervisor in the
+     form of a ROM.  The guest OS makes calls into this ROM to perform
+     privileged or sensitive actions in the system.
+
+  Extensibility.
+
+     In order to provide a vehicle for new features, new device support,
+     and general evolution, the API uses feature compartmentalization
+     with controlled versioning.  The API is split into sections, with
+     each section having independent versions.  Each section has a top
+     level version which is incremented for each major revision, with a
+     minor version indicating incremental level.  Version compatibility
+     is based on matching the major version field, and changes of the
+     major version are assumed to break compatibility.  This allows
+     accurate matching of compatibility.  In the event of incompatible
+     API changes, multiple APIs may be advertised by the hypervisor if it
+     wishes to support older versions of guest kernels.  This provides
+     the most general forward / backward compatibility possible.
+     Currently, the API has a core section for CPU / MMU virtualization
+     support, with additional sections provided for each supported device
+     class.
+
+2) Overview
+
+   Initialization.
+
+     Initialization is done with a bootstrap loader that creates
+     the "start of day" state.  This is a known state, running 32-bit
+     protected mode code with paging enabled.  The guest has all the
+     standard structures in memory that are provided by a native ROM
+     boot environment, including a memory map and ACPI tables.  For
+     the native hardware, this bootstrap loader can be run before
+     the kernel code proper, and this environment can be created
+     readily from within the hypervisor for the virtual case.  At
+     some point, the bootstrap loader or the kernel itself invokes
+     the initialization call to enter paravirtualized mode.
+
+   Privilege Model.
+
+     The guest kernel must be modified to run at a dynamic privilege
+     level, since if entry to paravirtual mode is successful, the kernel
+     is no longer allowed to run at the highest hardware privilege level.
+     On the IA-32 architecture, this means the kernel will be running at
+     CPL 1-2, and with the hypervisor running at CPL0, and user code at
+     CPL3.  The IOPL will be lowered as well to avoid giving the guest
+     direct access to hardware ports and control of the interrupt flag.
+
+     This change causes certain IA-32 instructions to become
"sensitive",
+     so additional support for clearing and setting the hardware
+     interrupt flag are present.  Since the switch into paravirtual mode
+     may happen dynamically, the guest OS must not rely on testing for a
+     specific privilege level by checking the RPL field of segment
+     selectors, but should check for privileged execution by performing
+     an (RPL != 3 && !EFLAGS_VM) comparison.  This means the DPL of
kernel
+     ring descriptors in the GDT or LDT may be raised to match the CPL of
+     the kernel.  This change is visible by inspecting the segments
+     registers while running in privileged code, and by using the LAR
+     instruction.
+
+     The system also cannot be allowed to write directly to the hardware
+     GDT, LDT, IDT, or TSS, so these data structures are maintained by the
+     hypervisor, and may be shadowed or guest visible structures.  These
+     structures are required to be page aligned to support non-shadowed
+     operation.
+
+     Currently, the system only provides for two guest security domains,
+     kernel (which runs at the equivalent of virtual CPL-0), and user
+     (which runs at the equivalent of virtual CPL-3, with no hardware
+     access).  Typically, this is not a problem, but if a guest OS relies
+     on using multiple hardware rings for privilege isolation, this
+     interface would need to be expanded to support that.
+
+   Memory Management.
+
+     Since a virtual machine typically does not have access to all the
+     physical memory on the machine, there is a need to redefine the
+     physical address space layout for the virtual machine.  The
+     spectrum of possibilities ranges from presenting the guest with
+     a view of a physically contiguous memory of a boot-time determined
+     size, exactly what the guest would see when running on hardware, to
+     the opposite, which presents the guest with the actual machine pages
+     which the hypervisor has allocated for it.  Using this approach
+     requires the guest to obtain information about the pages it has
+     from the hypervisor; this can be done by using the memory map which
+     would normally be passed to the guest by the BIOS.
+
+     The interface is designed to support either mode of operation.
+     This allows the implementation to use either direct page tables
+     or shadow page tables, or some combination of both.  All writes to
+     page table entries are done through calls to the hypervisor
+     interface layer.  The guest notifies the hypervisor about page
+     tables updates, flushes, and invalidations through API calls.
+
+     The guest OS is also responsible for notifying the hypervisor about
+     which pages in its physical memory are going to be used to hold page
+     tables or page directories.  Both PAE and non-PAE paging modes are
+     supported.  When the guest is finished using pages as page tables, it
+     should release them promptly to allow the hypervisor to free the
+     page table shadows.  Using a page as both a page table and a page
+     directory for linear page table access is possible, but currently
+     not supported by our implementation.
+
+     The hypervisor lives concurrently in the same address space as the
+     guest operating system.  Although this is not strictly necessary on
+     IA-32 hardware, performance would be severely degraded if that were
+     not the case.  The hypervisor must therefore reserve some portion of
+     linear address space for its own use. The implementation currently
+     reserves the top 64 megabytes of linear space for the hypervisor.
+     This requires the guest to relocate any data in high linear space
+     down by 64 megabytes.  For non-paging mode guests, this means the
+     high 64 megabytes of physical memory should be reserved.  Because
+     page tables are not sensitive to CPL, only to user/supervisor level,
+     the hypervisor must combine segment protection to ensure that the
+     guest can not access this 64 megabyte region.
+
+     An experimental patch is available to enable boot-time sizing of
+     the hypervisor hole.
+
+   Segmentation.
+
+     The IA-32 architecture provides segmented virtual memory, which can
+     be used as another form of privilege separation.  Each segment
+     contains a base, limit, and properties.  The base is added to the
+     virtual address to form a linear address.  The limit determines the
+     length of linear space which is addressable through the segment.
+     The properties determine read/write, code and data size of the
+     region, as well as the direction in which segments grow.  Segments
+     are loaded from descriptors in one of two system tables, the GDT or
+     the LDT, and the values loaded are cached until the next load of the
+     segment.  This property, known as segment caching, allows the
+     machine to be put into a non-reversible state by writing over the
+     descriptor table entry from which a segment was loaded.  There is no
+     efficient way to extract the base field of the segment after it is
+     loaded, as it is hidden by the processor.  In a hypervisor
+     environment, the guest OS can be interrupted at any point in time by
+     interrupts and NMIs which must be serviced by the hypervisor.  The
+     hypervisor must be able to recreate the original guest state when it
+     is done servicing the external event.
+
+     To avoid creating non-reversible segments, the hypervisor will
+     forcibly reload any live segment registers that are updated by
+     writes to the descriptor tables.  *N.B - in the event that a segment
+     is put into an invalid or not present state by an update to the
+     descriptor table, the segment register must be forced to NULL so
+     that reloading it will not cause a general protection fault (#GP)
+     when restoring the guest state.  This may require the guest to save
+     the segment register value before issuing a hypervisor API call
+     which will update the descriptor table.*
+
+     Because the hypervisor must protect its own memory space from
+     privileged code running in the guest at CPL1-2, descriptors may not
+     provide access to the 64 megabyte region of high linear space.  To
+     achieve this, the hypervisor will truncate descriptors in the
+     descriptor tables.  This means that attempts by the guest to access
+     through negative offsets to the segment base will fault, so this is
+     highly discouraged (some TLS implementations on Linux do this).
+     In addition, this causes the truncated length of the segment to
+     become visible to the guest through the LSL instruction.
+
+   Interrupt and I/O Subsystem.
+
+     For security reasons, the guest operating system is not given
+     control over the hardware interrupt flag.  We provide a virtual
+     interrupt flag that is under guest control.  The virtual operating
+     system always runs with hardware interrupts enabled, but hardware
+     interrupts are transparent to the guest.  The API provides calls for
+     all instructions which modify the interrupt flag.
+
+     The paravirtualization environment provides a legacy programmable
+     interrupt controller (PIC) to the virtual machine.  Future releases
+     will provide a virtual interrupt controller (VIC) that provides
+     more advanced features.
+
+     In addition to a virtual interrupt flag, there is also a virtual
+     IOPL field which the guest can use to enable access to port I/O
+     from userspace for privileged applications.
+
+     Generic PCI based device probing is available to detect virtual
+     devices.  The use of PCI is pragmatic, since it allows a vendor
+     ID, class ID, and device ID to identify the appropriate driver
+     for each virtual device.
+
+   IDT Management.
+
+     The paravirtual operating environment provides the traditional x86
+     interrupt descriptor table for handling external interrupts,
+     software interrupts, and exceptions.  The interrupt descriptor table
+     provides the destination code selector and EIP for interruptions.
+     The current task state structure (TSS) provides the new stack
+     address to use for interruptions that result in a privilege level
+     change.  The guest OS is responsible for notifying the hypervisor
+     when it updates the stack address in the TSS.
+
+     Two types of indirect control flow are of critical importance to the
+     performance of an operating system.  These are system calls and page
+     faults.  The guest is also responsible for calling out to the
+     hypervisor when it updates gates in the IDT.  Making IDT and TSS
+     updates known to the hypervisor in this fashion allows efficient
+     delivery through these performance critical gates.
+
+   Transparent Paravirtualization.
+
+     The guest operating system may provide an alternative implementation
+     of the VMI option rom compiled in.  This implementation should
+     provide implementations of the VMI calls that are suitable for
+     running on native x86 hardware.  This code may be used by the guest
+     operating system while it is being loaded, and may also be used if
+     the operating system is loaded on hardware that does not support
+     paravirtualization.
+
+     When the guest detects that the VMI option rom is available, it
+     replaces the compiled-in version of the rom with the rom provided by
+     the platform.  This can be accomplished by copying the rom contents,
+     or by remapping the virtual address containing the compiled-in rom
+     to point to the platform's ROM.  When booting on a platform that
+     does not provide a VMI rom, the operating system can continue to use
+     the compiled-in version to run in a non-paravirtualized fashion.
+
+   3rd Party Extensions.
+
+     If desired, it should be possible for 3rd party virtual machine
+     monitors to implement a paravirtualization environment that can run
+     guests written to this specification.
+
+     The general mechanism for providing customized features and
+     capabilities is to provide notification of these feature through
+     the CPUID call, and allowing configuration of CPU features
+     through RDMSR / WRMSR instructions.  This allows a hypervisor vendor
+     ID to be published, and the kernel may enable or disable specific
+     features based on this id.  This has the advantage of following
+     closely the boot time logic of many operating systems that enables
+     certain performance enhancements or bugfixes based on processor
+     revision, using exactly the same mechanism.
+
+     An exact formal specification of the new CPUID functions and which
+     functions are vendor specific is still needed.
+
+   AP Startup.
+
+     Application Processor startup in paravirtual SMP systems works a bit
+     differently than in a traditional x86 system.
+
+     APs will launch directly in paravirtual mode with initial state
+     provided by the BSP.  Rather than the traditional init/startup
+     IPI sequence, the BSP must issue the init IPI, a set application
+     processor state hypercall, followed by the startup IPI.
+
+     The initial state contains the AP's control registers, general
+     purpose registers and segment registers, as well as the IDTR,
+     GDTR, LDTR and EFER.  Any processor state not included in the initial
+     AP state (including x87 FPRs, SSE register states, and MSRs other than
+     EFER), are left in the poweron state.
+
+     The BSP must construct the initial GDT used by each AP.  The segment
+     register hidden state will be loaded from the GDT specified in the
+     initial AP state.  The IDT and (if used) LDT may either be constructed by
+     the BSP or by the AP.
+
+     Similarly, the initial page tables used by each AP must also be
+     constructed by the BSP.
+
+     If an AP's initial state is invalid, or no initial state is provided
+     before a start IPI is received by that AP, then the AP will fail to start.
+     It is therefore advisable to have a timeout for waiting for AP's to
start,
+     as is recommended for traditional x86 systems.
+
+     See VMI_SetInitialAPState in Appendix A for a description of the
+     VMI_SetInitialAPState hypercall and the associated APState data structure.
+
+   State Synchronization In SMP Systems.
+
+     Some in-memory data structures that may require no special synchronization
+     on a traditional x86 systems need special handling when run on a
+     hypervisor.  Two of particular note are the descriptor tables and page
+     tables.
+
+     Each processor in an SMP system should have its own GDT and LDT.  Changes
+     to each processor's descriptor tables must be made on that processor
+     via the appropriate VMI calls.  There is no VMI interface for updating
+     another CPU's descriptor tables (aside from VMI_SetInitialAPState),
+     and the result of memory writes to other processors' descriptor tables
+     are undefined.
+
+     Page tables have slightly different semantics than in a traditional x86
+     system.  As in traditional x86 systems, page table writes may not be
+     respected by the current CPU until a TLB flush or invlpg is issued.
+     In a paravirtual system, the hypervisor implementation is free to
+     provide either shared or private caches of the guest's page tables.
+     Page table updates must therefore be propagated to the other CPUs
+     before they are guaranteed to be noticed.
+
+     In particular, when doing TLB shootdown, the initiating processor
+     must ensure that all deferred page table updates are flushed to the
+     hypervisor, to ensure that the receiving processor has the most up-to-date
+     mapping when it performs its invlpg.
+
+   Local APIC Support.
+
+     A traditional x86 local APIC is provided by the hypervisor.  The local
+     APIC is enabled and its address is set via the IA32_APIC_BASE MSR, as
+     usual.  APIC registers may be read and written via ordinary memory
+     operations.
+
+     For performance reasons, higher performance APIC read and write interfaces
+     are provided.  If possible, these interfaces should be used to access
+     the local APIC.
+
+     The IO-APIC is not included in this spec, as it is typically not
+     performance critical, and used mainly for initial wiring of IRQ pins.
+     Currently, we implement a fully functional IO-APIC with all the
+     capabilities of real hardware.  This may seem like an unnecessary burden,
+     but if the goal is transparent paravirtualization, the kernel must
+     provide fallback support for an IO-APIC anyway.  In addition, the
+     hypervisor must support an IO-APIC for SMP non-paravirtualized guests.
+     The net result is less code on both sides, and an already well defined
+     interface between the two.  This avoids the complexity burden of having
+     to support two different interfaces to achieve the same task.
+
+     One shortcut we have found most helpful is to simply disable NMI delivery
+     to the paravirtualized kernel.  There is no reason NMIs can't be
+     supported, but typical uses for them are not as productive in a
+     virtualized environment.  Watchdog NMIs are of limited use if the OS is
+     already correct and running on stable hardware; profiling NMIs are
+     similarly of less use, since this task is accomplished with more accuracy
+     in the VMM itself; and NMIs for machine check errors should be handled
+     outside of the VM.  The addition of NMI support does create additional
+     complexity for the trap handling code in the VM, and although the task is
+     surmountable, the value proposition is debatable.  Here, again, feedback
+     is desired.
+
+   Time Interface.
+
+     In a virtualized environment, virtual machines (VM) will time share
+     the system with each other and with other processes running on the
+     host system.  Therefore, a VM's virtual CPUs (VCPUs) will be
+     executing on the host's physical CPUs (PCPUs) for only some portion
+     of time.  This section of the VMI exposes a paravirtual view of
+     time to the guest operating systems so that they may operate more
+     effectively in a virtual environment.  The interface also provides
+     a way for the VCPUs to set alarms in this paravirtual view of time.
+
+     Time Domains:
+
+     a) Wallclock Time:
+
+     Wallclock time exposed to the VM through this interface indicates
+     the number of nanoseconds since epoch, 1970-01-01T00:00:00Z (ISO
+     8601 date format).  If the host's wallclock time changes (say, when
+     an error in the host's clock is corrected), so does the wallclock
+     time as viewed through this interface.
+
+     b) Real Time:
+
+     Another view of time accessible through this interface is real
+     time.  Real time always progresses except for when the VM is
+     stopped or suspended.  Real time is presented to the guest as a
+     counter which increments at a constant rate defined (and presented)
+     by the hypervisor.  All the VCPUs of a VM share the same real time
+     counter.
+
+     The unit of the counter is called "cycles".  The unit and
initial
+     value (corresponding to the time the VM enters para-virtual mode)
+     are chosen by the hypervisor so that the real time counter will not
+     rollover in any practical length of time.  It is expected that the
+     frequency (cycles per second) is chosen such that this clock
+     provides a "high-resolution" view of time.  The unit can only
+     change when the VM (re)enters paravirtual mode.
+
+     c) Stolen time and Available time:
+
+     A VCPU is always in one of three states: running, halted, or ready.
+     The VCPU is in the 'running' state if it is executing.  When the
+     VCPU executes the HLT interface, the VCPU enters the 'halted'
state
+     and remains halted until there is some work pending for the VCPU
+     (e.g. an alarm expires, host I/O completes on behalf of virtual
+     I/O).  At this point, the VCPU enters the 'ready' state (waiting
+     for the hypervisor to reschedule it).  Finally, at any time when
+     the VCPU is not in the 'running' state nor the 'halted'
state, it
+     is in the 'ready' state.
+
+     For example, consider the following sequence of events, with times
+     given in real time:
+
+     (Example 1)
+
+     At 0 ms, VCPU executing guest code.
+     At 1 ms, VCPU requests virtual I/O.
+     At 2 ms, Host performs I/O for virtual I/0.
+     At 3 ms, VCPU executes VMI_Halt.
+     At 4 ms, Host completes I/O for virtual I/O request.
+     At 5 ms, VCPU begins executing guest code, vectoring to the interrupt
+              handler for the device initiating the virtual I/O.
+     At 6 ms, VCPU preempted by hypervisor.
+     At 9 ms, VCPU begins executing guest code.
+
+     From 0 ms to 3 ms, VCPU is in the 'running' state.  At 3 ms, VCPU
+     enters the 'halted' state and remains in this state until the 4 ms
+     mark.  From 4 ms to 5 ms, the VCPU is in the 'ready' state.  At 5
+     ms, the VCPU re-enters the 'running' state until it is preempted
by
+     the hypervisor at the 6 ms mark.  From 6 ms to 9 ms, VCPU is again
+     in the 'ready' state, and finally 'running' again after 9
ms.
+
+     Stolen time is defined per VCPU to progress at the rate of real
+     time when the VCPU is in the 'ready' state, and does not progress
+     otherwise.  Available time is defined per VCPU to progress at the
+     rate of real time when the VCPU is in the 'running' and
'halted'
+     states, and does not progress when the VCPU is in the 'ready'
+     state.
+
+     So, for the above example, the following table indicates these time
+     values for the VCPU at each ms boundary:
+
+     Real time    Stolen time    Available time
+      0            0              0
+      1            0              1
+      2            0              2
+      3            0              3
+      4            0              4
+      5            1              4
+      6            1              5
+      7            2              5
+      8            3              5
+      9            4              5
+     10            4              6
+
+     Notice that at any point:
+        real_time == stolen_time + available_time
+
+     Stolen time and available time are also presented as counters in
+     "cycles" units.  The initial value of the stolen time counter is
0.
+     This implies the initial value of the available time counter is the
+     same as the real time counter.
+
+     Alarms:
+
+     Alarms can be set (armed) against the real time counter or the
+     available time counter. Alarms can be programmed to expire once
+     (one-shot) or on a regular period (periodic).  They are armed by
+     indicating an absolute counter value expiry, and in the case of a
+     periodic alarm, a non-zero relative period counter value.  [TBD:
+     The method of wiring the alarms to an interrupt vector is dependent
+     upon the virtual interrupt controller portion of the interface.
+     Currently, the alarms may be wired as if they are attached to IRQ0
+     or the vector in the local APIC LVTT.  This way, the alarms can be
+     used as drop in replacements for the PIT or local APIC timer.]
+
+     Alarms are per-vcpu mechanisms.  An alarm set by vcpu0 will fire
+     only on vcpu0, while an alarm set by vcpu1 will only fire on vcpu1.
+     If an alarm is set relative to available time, its expiry is a
+     value relative to the available time counter of the vcpu that set
+     it.
+
+     The interface includes a method to cancel (disarm) an alarm.  On
+     each vcpu, one alarm can be set against each of the two counters
+     (real time and available time).  A vcpu in the 'halted' state
+     becomes 'ready' when any of its alarm's counters reaches the
+     expiry.
+
+     An alarm "fires" by signaling the virtual interrupt controller.
An
+     alarm will fire as soon as possible after the counter value is
+     greater than or equal to the alarm's current expiry.  However, an
+     alarm can fire only when its vcpu is in the 'running' state.
+
+     If the alarm is periodic, a sequence of expiry values,
+
+      E(i) = e0 + p * i ,  i = 0, 1, 2, 3, ...
+
+     where 'e0' is the expiry specified when setting the alarm and
'p'
+     is the period of the alarm, is used to arm the alarm.  Initially,
+     E(0) is used as the expiry.  When the alarm fires, the next expiry
+     value in the sequence that is greater than the current value of the
+     counter is used as the alarm's new expiry.
+
+     One-shot alarms have only one expiry.  When a one-shot alarm fires,
+     it is automatically disarmed.
+
+     Suppose an alarm is set relative to real time with expiry at the 3
+     ms mark and a period of 2 ms.  It will expire on these real time
+     marks: 3, 5, 7, 9.  Note that even if the alarm does not fire
+     during the 5 ms to 7 ms interval, the alarm can fire at most once
+     during the 7 ms to 9 ms interval (unless, of course, it is
+     reprogrammed).
+
+     If an alarm is set relative to available time with expiry at the 1
+     ms mark (in available time) and with a period of 2 ms, then it will
+     expire on these available time marks: 1, 3, 5.  In the scenario
+     described in example 1, those available time values correspond to
+     these values in real time: 1, 3, 6.
+
+3) Architectural Differences from Native Hardware.
+
+     For the sake of performance, some requirements are imposed on kernel
+     fault handlers which are not present on real hardware.  Most modern
+     operating systems should have no trouble meeting these requirements.
+     Failure to meet these requirements may prevent the kernel from
+     working properly.
+
+     1) The hardware flags on entry to a fault handler may not match
+        the EFLAGS image on the fault handler stack.  The stack image
+        is correct, and will have the correct state of the interrupt
+        and arithmetic flags.
+
+     2) The stack used for kernel traps must be flat - that is, zero base,
+        segment limit determined by the hypervisor.
+
+     3) On entry to any fault handler, the stack must have sufficient space
+        to hold 32 bytes of data, or the guest may be terminated.
+
+     4) When calling VMI functions, the kernel must be running on a
+        flat 32-bit stack and code segment.
+
+     5) Most VMI functions require flat data and extra segment (DS and ES)
+        segments as well; notable exceptions are IRET and SYSEXIT.
+        XXXPara - may need to add STI and CLI to this list.
+
+     6) Interrupts must always be enabled when running code in userspace.
+
+     7) IOPL semantics for userspace are changed; although userspace may be
+        granted port access, it can not affect the interrupt flag.
+
+     8) The EIPs at which faults may occur in VMI calls may not match the
+        original native instruction EIP; this is a bug in the system
+        today, as many guests do rely on lazy fault handling.
+
+     9) On entry to V8086 mode, MSR_SYSENTER_CS is cleared to zero.
+
+     10) Todo - we would like to support these features, but they are not
+        fully tested and / or implemented:
+
+        Userspace 16-bit stack support
+        Proper handling of faulting IRETs
+
+4) ROM Implementation
+
+   Modularization
+
+     Originally, we envisioned modularizing the ROM API into several
+     subsections, but the close coupling between the initial layers
+     and the requirement to support native PCI bus devices has made
+     ROM components for network or block devices unnecessary to this
+     point in time.
+
+	VMI - the virtual machine interface.  This is the core CPU, I/O
+	      and MMU virtualization layer.  I/O is currently limited
+              to port access to emulated devices.
+
+   Detection
+
+      The presence of hypervisor ROMs can be recognized by scanning the
+      upper region of the first megabyte of physical memory.  Multiple
+      ROMs may be provided to support older API versions for legacy guest
+      OS support.  ROM detection is done in the traditional manner, by
+      scanning the memory region from C8000h - DFFFFh in 2 kilobyte
+      increments.  The romSignature bytes must be '0x55, 0xAA', and the
+      checksum of the region indicated by the romLength field must be zero.
+      The checksum is a simple 8-bit addition of all bytes in the ROM region.
+
+   Data layout
+
+      typedef struct HyperRomHeader {
+         uint16_t        romSignature;
+         int8_t          romLength;
+         unsigned char   romEntry[4];
+         uint8_t         romPad0;
+         uint32_t        hyperSignature;
+         uint8_t         APIVersionMinor;
+         uint8_t         APIVersionMajor;
+         uint8_t         reserved0;
+         uint8_t         reserved1;
+         uint32_t        reserved2;
+         uint32_t        reserved3;
+         uint16_t        pciHeaderOffset;
+         uint16_t        pnpHeaderOffset;
+         uint32_t        romPad3;
+         char            reserved[32];
+         char            elfHeader[64];
+      } HyperRomHeader;
+
+      The first set of fields is defined by the BIOS:
+
+      romSignature - fixed 0xAA55, BIOS ROM signature
+      romLength    - the length of the ROM, in 512 byte chunks.
+                     Determines the area to be checksummed.
+      romEntry     - 16-bit initialization code stub used by BIOS.
+      romPad0      - reserved
+
+      The next set of fields is defined by this API:
+
+      hyperSignature  - a 4 byte signature providing recognition of the
+	                device class represented by this ROM.  Each
+	                device class defines its own unique signature.
+      APIVersionMinor - the revision level of this device class' API.
+	                This indicates incremental changes to the API.
+      APIVersionMajor - the major version. Used to indicates large
+	                revisions or additions to the API which break
+	                compatibility with the previous version.
+      reserved0,1,2,3 - for future expansion
+
+      The next set of fields is defined by the PCI / PnP BIOS spec:
+
+      pciHeaderOffset - relative offset to the PCI device header from
+	  	        the start of this ROM.
+      pnpHeaderOffset - relative offset to the PnP boot header from the
+	                start of this ROM.
+      romPad3         - reserved by PCI spec.
+
+      Finally, there is space for future header fields, and an area
+      reserved for an ELF header to point to symbol information.
+
+Appendix A - VMI ROM Low Level ABI
+
+   OS writers intending to port their OS to the paravirtualizable x86
+   processor being modeled by this hypervisor need to access the
+   hypervisor through the VMI layer. It is possible although it is
+   currently unimplemented to add or replace the functionality of
+   individual hypervisor calls by providing your own ROM images. This is
+   intended to allow third party customizations.
+
+   VMI compatible ROMs user the signature "cVmi" in the
hyperSignature
+   field of the ROM header.
+
+   Many of these calls are compatible with the SVR4 C call ABI, using up
+   to three register arguments.   Some calls are not, due to restrictions
+   of the native instruction set.  Calls which diverge from this ABI are
+   noted.  In GNU terms, this means most of the calls are compatible with
+   regparm(3) argument passing.
+
+   Most of these calls behave as standard C functions, and as such, may
+   clobber registers EAX, EDX, ECX, flags.  Memory clobbers are noted
+   explicitly, since many of them may be inlined without a memory clobber.
+
+   Most of these calls require well defined segment conventions - that is,
+   flat full size 32-bit segments for all the general segments, CS, SS, DS,
+   ES.  Exceptions in some cases are noted.
+
+   The net result of these choices is that most of the calls are very
+   easy to make from C-code, and calls that are likely to be required in
+   low level trap handling code are easy to call from assembler.   Most
+   of these calls are also very easily implemented by the hypervisor
+   vendor in C code, and only the performance critical calls from
+   assembler paths require custom assembly implementations.
+
+   CORE INTERFACE CALLS
+
+    This set of calls provides the base functionality to establish running
+    the kernel in VMI mode.
+
+    The interface will be expanded to include feature negotiation, more
+    explicit control over call bundling and flushing, and hypervisor
+    notifications to allow inline code patching.
+
+    VMI_Init
+
+       VMICALL void VMI_Init(void);
+
+       Initializes the hypervisor environment.  Returns zero on success,
+       or -1 if the hypervisor could not be initialized.  Note that this
+       is a recoverable error if the guest provides the requisite native
+       code to support transparent paravirtualization.
+
+       Inputs:      None
+       Outputs:     EAX = result
+       Clobbers:    Standard
+       Segments:    Standard
+
+
+   PROCESSOR STATE CALLS
+
+    This set of calls controls the online status of the processor.  It
+    include interrupt control, reboot, halt, and shutdown functionality.
+    Future expansions may include deep sleep and hotplug CPU capabilities.
+
+    VMI_DisableInterrupts
+
+       VMICALL void VMI_DisableInterrupts(void);
+
+       Disable maskable interrupts on the processor.
+
+       Inputs:      None
+       Outputs:     None
+       Clobbers:    Flags only
+       Segments:    As this is both performance critical and likely to
+          be called from low level interrupt code, this call does not
+          require flat DS/ES segments, but uses the stack segment for
+          data access.  Therefore only CS/SS must be well defined.
+
+    VMI_EnableInterrupts
+
+       VMICALL void VMI_EnableInterrupts(void);
+
+       Enable maskable interrupts on the processor.  Note that the
+       current implementation always will deliver any pending interrupts
+       on a call which enables interrupts, for compatibility with kernel
+       code which expects this behavior.  Whether this should be required
+       is open for debate.
+
+       Inputs:      None
+       Outputs:     None
+       Clobbers:    Flags only
+       Segments:    CS/SS only
+
+    VMI_GetInterruptMask
+
+       VMICALL VMI_UINT VMI_GetInterruptMask(void);
+
+       Returns the current interrupt state mask of the processor.  The
+       mask is defined to be 0x200 (matching processor flag IF) to indicate
+       interrupts are enabled.
+
+       Inputs:      None
+       Outputs:     EAX = mask
+       Clobbers:    Flags only
+       Segments:    CS/SS only
+
+    VMI_SetInterruptMask
+
+       VMICALL void VMI_SetInterruptMask(VMI_UINT mask);
+
+       Set the current interrupt state mask of the processor.  Also
+       delivers any pending interrupts if the mask is set to allow
+       them.
+
+       Inputs:      EAX = mask
+       Outputs:     None
+       Clobbers:    Flags only
+       Segments:    CS/SS only
+
+    VMI_DeliverInterrupts (For future debate)
+
+       Enable and deliver any pending interrupts.  This would remove
+       the implicit delivery semantic from the SetInterruptMask and
+       EnableInterrupts calls.
+
+    VMI_Pause
+
+       VMICALL void VMI_Pause(void);
+
+       Pause the processor temporarily, to allow a hypertwin or remote
+       CPU to continue operation without lock or cache contention.
+
+       Inputs:      None
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_Halt
+
+       VMICALL void VMI_Halt(void);
+
+       Put the processor into interruptible halt mode.  This is defined
+       to be a non-running mode where maskable interrupts are enabled,
+       not a deep low power sleep mode.
+
+       Inputs:      None
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_Shutdown
+
+       VMICALL void VMI_Shutdown(void);
+
+       Put the processor into non-interruptible halt mode.  This is defined
+       to be a non-running mode where maskable interrupts are disabled,
+       indicates a power-off event for this CPU.
+
+       Inputs:      None
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_Reboot:
+
+       VMICALL void VMI_Reboot(VMI_INT how);
+
+       Reboot the virtual machine, using a hard or soft reboot.  A soft
+       reboot corresponds to the effects of an INIT IPI, and preserves
+       some APIC and CR state.  A hard reboot corresponds to a hardware
+       reset.
+
+       Inputs:      EAX = reboot mode
+                      #define VMI_REBOOT_SOFT 0x0
+                      #define VMI_REBOOT_HARD 0x1
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+   VMI_SetInitialAPState:
+
+       void VMI_SetInitialAPState(APState *apState, VMI_UINT32 apicID);
+
+       Sets the initial state of the application processor with local APIC ID
+       "apicID" to the state in apState.  apState must be the
page-aligned
+       linear address of the APState structure describing the initial state of
+       the specified application processor.
+
+       Control register CR0 must have both PE and PG set;  the result of
+       either of these bits being cleared is undefined.  It is recommended
+       that for best performance, all processors in the system have the same
+       setting of the CR4 PAE bit.  LME and LMA in EFER are both currently
+       unsupported.  The result of setting either of these bits is undefined.
+
+       Inputs:      EAX = pointer to APState structure for new co-processor
+                    EDX = APIC ID of processor to initialize
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+
+   DESCRIPTOR RELATED CALLS
+
+    VMI_SetGDT
+
+       VMICALL void VMI_SetGDT(VMI_DTR *gdtr);
+
+       Load the global descriptor table limit and base registers.  In
+       addition to the straightforward load of the hardware registers, this
+       has the additional side effect of reloading all segment registers in a
+       virtual machine.  The reason is that otherwise, the hidden part of
+       segment registers (the base field) may be put into a non-reversible
+       state.  Non-reversible segments are problematic because they can not be
+       reloaded - any subsequent loads of the segment will load the new
+       descriptor state.  In general, is not possible to resume direct
+       execution of the virtual machine if certain segments become
+       non-reversible.
+
+       A load of the GDTR may cause the guest visible memory image of the GDT
+       to be changed.  This allows the hypervisor to share the GDT pages with
+       the guest, but also continue to maintain appropriate protections on the
+       GDT page by transparently adjusting the DPL and RPL of descriptors in
+       the GDT.
+
+       Inputs:      EAX = pointer to descriptor limit / base
+       Outputs:     None
+       Clobbers:    Standard, Memory
+       Segments:    Standard
+
+    VMI_SetIDT
+
+       VMICALL void VMI_SetIDT(VMI_DTR *idtr);
+
+       Load the interrupt descriptor table limit and base registers.  The IDT
+       format is defined to be the same as native hardware.
+
+       A load of the IDTR may cause the guest visible memory image of the IDT
+       to be changed.  This allows the hypervisor to rewrite the IDT pages in
+       a format more suitable to the hypervisor, which may include adjusting
+       the DPL and RPL of descriptors in the guest IDT.
+
+       Inputs:      EAX = pointer to descriptor limit / base
+       Outputs:     None
+       Clobbers:    Standard, Memory
+       Segments:    Standard
+
+    VMI_SetLDT
+
+       VMICALL void VMI_SetLDT(VMI_SELECTOR ldtSel);
+
+       Load the local descriptor table.  This has the additional side effect
+       of of reloading all segment registers.  See VMI_SetGDT for an
+       explanation of why this is required.  A load of the LDT may cause the
+       guest visible memory image of the LDT to be changed, just as GDT and
+       IDT loads.
+
+       Inputs:      EAX = GDT selector of LDT descriptor
+       Outputs:     None
+       Clobbers:    Standard, Memory
+       Segments:    Standard
+
+    VMI_SetTR
+
+       VMICALL void VMI_SetTR(VMI_SELECTOR ldtSel);
+
+       Load the task register.  Functionally equivalent to the LTR
+       instruction.
+
+       Inputs:      EAX = GDT selector of TR descriptor
+       Outputs:     None
+       Clobbers:    Standard, Memory
+       Segments:    Standard
+
+    VMI_GetGDT
+
+       VMICALL void VMI_GetGDT(VMI_DTR *gdtr);
+
+       Copy the GDT limit and base fields into the provided pointer.  This is
+       equivalent to the SGDT instruction, which is non-virtualizable.
+
+       Inputs:      EAX = pointer to descriptor limit / base
+       Outputs:     None
+       Clobbers:    Standard, Memory
+       Segments:    Standard
+
+    VMI_GetIDT
+
+       VMICALL void VMI_GetIDT(VMI_DTR *idtr);
+
+       Copy the IDT limit and base fields into the provided pointer.  This is
+       equivalent to the SIDT instruction, which is non-virtualizable.
+
+       Inputs:      EAX = pointer to descriptor limit / base
+       Outputs:     None
+       Clobbers:    Standard, Memory
+       Segments:    Standard
+
+    VMI_GetLDT
+
+       VMICALL VMI_SELECTOR VMI_GetLDT(void);
+
+       Load the task register.  Functionally equivalent to the SLDT
+       instruction, which is non-virtualizable.
+
+       Inputs:      None
+       Outputs:     EAX = selector of LDT descriptor
+       Clobbers:    Standard, Memory
+       Segments:    Standard
+
+    VMI_GetTR
+
+       VMICALL VMI_SELECTOR VMI_GetTR(void);
+
+       Load the task register.  Functionally equivalent to the STR
+       instruction, which is non-virtualizable.
+
+       Inputs:      None
+       Outputs:     EAX = selector of TR descriptor
+       Clobbers:    Standard, Memory
+       Segments:    Standard
+
+    VMI_WriteGDTEntry
+
+       VMICALL void VMI_WriteGDTEntry(void *gdt, VMI_UINT entry,
+                                      VMI_UINT32 descLo,
+                                      VMI_UINT32 descHi);
+
+       Write a descriptor to a GDT entry.  Note that writes to the GDT itself
+       may be disallowed by the hypervisor, in which case this call must be
+       converted into a hypercall.  In addition, since the descriptor may need
+       to be modified to change limits and / or permissions, the guest kernel
+       should not assume the update will be binary identical to the passed
+       input.
+
+       Inputs:      EAX   = pointer to GDT base
+                    EDX   = GDT entry number
+                    ECX   = descriptor low word
+                    ST(1) = descriptor high word
+       Outputs:     None
+       Clobbers:    Standard, Memory
+       Segments:    Standard
+
+    VMI_WriteLDTEntry
+
+       VMICALL void VMI_WriteLDTEntry(void *gdt, VMI_UINT entry,
+                                      VMI_UINT32 descLo,
+                                      VMI_UINT32 descHi);
+
+       Write a descriptor to a LDT entry.  Note that writes to the LDT itself
+       may be disallowed by the hypervisor, in which case this call must be
+       converted into a hypercall.  In addition, since the descriptor may need
+       to be modified to change limits and / or permissions, the guest kernel
+       should not assume the update will be binary identical to the passed
+       input.
+
+       Inputs:      EAX   = pointer to LDT base
+                    EDX   = LDT entry number
+                    ECX   = descriptor low word
+                    ST(1) = descriptor high word
+       Outputs:     None
+       Clobbers:    Standard, Memory
+       Segments:    Standard
+
+    VMI_WriteIDTEntry
+
+       VMICALL void VMI_WriteIDTEntry(void *gdt, VMI_UINT entry,
+                                      VMI_UINT32 descLo,
+                                      VMI_UINT32 descHi);
+
+       Write a descriptor to a IDT entry.  Since the descriptor may need to be
+       modified to change limits and / or permissions, the guest kernel should
+       not assume the update will be binary identical to the passed input.
+
+       Inputs:      EAX   = pointer to IDT base
+                    EDX   = IDT entry number
+                    ECX   = descriptor low word
+                    ST(1) = descriptor high word
+       Outputs:     None
+       Clobbers:    Standard, Memory
+       Segments:    Standard
+
+
+   CPU CONTROL CALLS
+
+    These calls encapsulate the set of privileged instructions used to
+    manipulate the CPU control state.  These instructions are all properly
+    virtualizable using trap and emulate, but for performance reasons, a
+    direct call may be more efficient.  With hardware virtualization
+    capabilities, many of these calls can be left as IDENT translations, that
+    is, inline implementations of the native instructions, which are not
+    rewritten by the hypervisor.  Some of these calls are performance critical
+    during context switch paths, and some are not, but they are all included
+    for completeness, with the exceptions of the obsoleted LMSW and SMSW
+    instructions.
+
+    VMI_WRMSR
+
+       VMICALL void VMI_WRMSR(VMI_UINT64 val, VMI_UINT32 reg);
+
+       Write to a model specific register.  This functions identically to the
+       hardware WRMSR instruction.  Note that a hypervisor may not implement
+       the full set of MSRs supported by native hardware, since many of them
+       are not useful in the context of a virtual machine.
+
+       Inputs:      ECX = model specific register index
+                    EAX = low word of register
+                    EDX = high word of register
+       Outputs:     None
+       Clobbers:    Standard, Memory
+       Segments:    Standard
+
+    VMI_RDMSR
+
+       VMICALL VMI_UINT64 VMI_RDMSR(VMI_UINT64 dummy, VMI_UINT32 reg);
+
+       Read from a model specific register.  This functions identically to the
+       hardware RDMSR instruction.  Note that a hypervisor may not implement
+       the full set of MSRs supported by native hardware, since many of them
+       are not useful in the context of a virtual machine.
+
+       Inputs:      ECX = machine specific register index
+       Outputs:     EAX = low word of register
+                    EDX = high word of register
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_SetCR0
+
+       VMICALL void VMI_SetCR0(VMI_UINT val);
+
+       Write to control register zero.  This can cause TLB flush and FPU
+       handling side effects.  The set of features available to the kernel
+       depend on the completeness of the hypervisor.  An explicit list of
+       supported functionality or required settings may need to be negotiated
+       by the hypervisor and kernel during bootstrapping.  This is likely to
+       be implementation or vendor specific, and the precise restrictions are
+       not yet worked out.  Our implementation in general supports turning on
+       additional functionality - enabling protected mode, paging, page write
+       protections; however, once those features have been enabled, they may
+       not be disabled on the virtual hardware.
+
+       Inputs:      EAX = input to control register
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_SetCR2
+
+       VMICALL void VMI_SetCR2(VMI_UINT val);
+
+       Write to control register two.  This has no side effects other than
+       updating the CR2 register value.
+
+       Inputs:      EAX = input to control register
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_SetCR3
+
+       VMICALL void VMI_SetCR3(VMI_UINT val);
+
+       Write to control register three.  This causes a TLB flush on the local
+       processor.  In addition, this update may be queued as part of a lazy
+       call invocation, which allows multiple hypercalls to be issued during
+       the context switch path.  The queuing convention is to be negotiated
+       with the hypervisor during bootstrapping, but the interfaces for this
+       negotiation are currently vendor specific.
+
+       Inputs:      EAX = input to control register
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+       Queue Class: MMU
+
+    VMI_SetCR4
+
+       VMICALL void VMI_SetCR3(VMI_UINT val);
+
+       Write to control register four.  This can cause TLB flush and many
+       other CPU side effects.  The set of features available to the kernel
+       depend on the completeness of the hypervisor.  An explicit list of
+       supported functionality or required settings may need to be negotiated
+       by the hypervisor and kernel during bootstrapping.  This is likely to
+       be implementation or vendor specific, and the precise restrictions are
+       not yet worked out.  Our implementation in general supports turning on
+       additional MMU functionality - enabling global pages, large pages, PAE
+       mode, and other features - however, once those features have been
+       enabled, they may not be disabled on the virtual hardware.  The
+       remaining CPU control bits of CR4 remain active and behave identically
+       to real hardware.
+
+       Inputs:      EAX = input to control register
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_GetCR0
+    VMI_GetCR2
+    VMI_GetCR3
+    VMI_GetCR4
+
+       VMICALL VMI_UINT32 VMI_GetCR0(void);
+       VMICALL VMI_UINT32 VMI_GetCR2(void);
+       VMICALL VMI_UINT32 VMI_GetCR3(void);
+       VMICALL VMI_UINT32 VMI_GetCR4(void);
+
+       Read the value of a control register into EAX.  The register contents
+       are identical to the native hardware control registers; CR0 contains
+       the control bits and task switched flag, CR2 contains the last page
+       fault address, CR3 contains the page directory base pointer, and CR4
+       contains various feature control bits.
+
+       Inputs:      None
+       Outputs:     EAX = value of control register
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_CLTS
+
+       VMICALL void VMI_CLTS(void);
+
+       Used to clear the task switched (TS) flag in control register zero.  A
+       replacement for the CLTS instruction.
+
+       Inputs:      None
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+     VMI_SetDR
+
+       VMICALL void VMI_SetDR(VMI_UINT32 num, VMI_UINT32 val);
+
+       Set the debug register to the given value.  If a hypervisor
+       implementation supports debug registers, this functions equivalently to
+       native hardware move to DR instructions.
+
+       Inputs:      EAX = debug register number
+                    EDX = debug register value
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+     VMI_GetDR
+
+       VMICALL VMI_UINT32 VMI_GetDR(VMI_UINT32 num);
+
+       Read a debug register.  If debug registers are not supported, the
+       implementation is free to return zero values.
+
+       Inputs:      EAX = debug register number
+       Outputs:     EAX = debug register value
+       Clobbers:    Standard
+       Segments:    Standard
+
+
+   PROCESSOR INFORMATION CALLS
+
+    These calls provide access to processor identification, performance and
+    cycle data, which may be inaccurate due to the nature of running on
+    virtual hardware.   This information may be visible in a non-virtualizable
+    way to applications running outside of the kernel.  As such, both RDTSC
+    and RDPMC should be disabled by kernels or hypervisors where information
+    leakage is a concern, and the accuracy of data retrieved by these functions
+    is up to the individual hypervisor vendor.
+
+    VMI_CPUID
+
+       /* Not expressible as a C function */
+
+       The CPUID instruction provides processor feature identification in a
+       vendor specific manner.  The instruction itself is non-virtualizable
+       without hardware support, requiring a hypervisor assisted CPUID call
+       that emulates the effect of the native instruction, while masking any
+       unsupported CPU feature bits.
+
+       Inputs:       EAX = CPUID number
+                     ECX = sub-level query (nonstandard)
+       Outputs:      EAX = CPUID dword 0
+                     EBX = CPUID dword 1
+                     ECX = CPUID dword 2
+                     EDX = CPUID dword 3
+       Clobbers:     Flags only
+       Segments:     Standard
+
+    VMI_RDTSC
+
+       VMICALL VMI_UINT64 VMI_RDTSC(void);
+
+       The RDTSC instruction provides a cycles counter which may be made
+       visible to userspace.  For better or worse, many applications have made
+       use of this feature to implement userspace timers, database indices, or
+       for micro-benchmarking of performance.  This instruction is extremely
+       problematic for virtualization, because even though it is selectively
+       virtualizable using trap and emulate, it is much more expensive to
+       virtualize it in this fashion.  On the other hand, if this instruction
+       is allowed to execute without trapping, the cycle counter provided
+       could be wrong in any number of circumstances due to hardware drift,
+       migration, suspend/resume, CPU hotplug, and other unforeseen
+       consequences of running inside of a virtual machine.  There is no
+       standard specification for how this instruction operates when issued
+       from userspace programs, but the VMI call here provides a proper
+       interface for the kernel to read this cycle counter.
+
+       Inputs:      None
+       Outputs:     EAX = low word of TSC cycle counter
+                    EDX = high word of TSC cycle counter
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_RDPMC
+
+       VMICALL VMI_UINT64 VMI_RDPMC(VMI_UINT64 dummy, VMI_UINT32 counter);
+
+       Similar to RDTSC, this call provides the functionality of reading
+       processor performance counters.  It also is selectively visible to
+       userspace, and maintaining accurate data for the performance counters
+       is an extremely difficult task due to the side effects introduced by
+       the hypervisor.
+
+       Inputs:      ECX = performance counter index
+       Outputs:     EAX = low word of counter
+                    EDX = high word of counter
+       Clobbers:    Standard
+       Segments:    Standard
+
+
+   STACK / PRIVILEGE TRANSITION CALLS
+
+    This set of calls encapsulates mechanisms required to transfer between
+    higher privileged kernel tasks and userspace.  The stack switching and
+    return mechanisms are also used to return from interrupt handlers into
+    the kernel, which may involve atomic interrupt state and stack
+    transitions.
+
+    VMI_UpdateKernelStack
+
+       VMICALL void VMI_UpdateKernelStack(void *tss, VMI_UINT32 esp0);
+
+       Inform the hypervisor that a new kernel stack pointer has been loaded
+       in the TSS structure.  This new kernel stack pointer will be used for
+       entry into the kernel on interrupts from userspace.
+
+       Inputs:      EAX = pointer to TSS structure
+                    EDX = new kernel stack top
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_IRET
+
+       /* No C prototype provided */
+
+       Perform a near equivalent of the IRET instruction, which atomically
+       switches off the current stack and restore the interrupt mask.  This
+       may return to userspace or back to the kernel from an interrupt or
+       exception handler.  The VMI_IRET call does not restore IOPL from the
+       stack image, as the native hardware equivalent would.  Instead, IOPL
+       must be explicitly restored using a VMI_SetIOPL call.  The VMI_IRET
+       call does, however, restore the state of the EFLAGS_VM bit from the
+       stack image in the event that the hypervisor and kernel both support
+       V8086 execution mode.  If the hypervisor does not support V8086 mode,
+       this can be silently ignored, generating an error that the guest must
+       deal with.  Note this call is made using a CALL instruction, just as
+       all other VMI calls, so the EIP of the call site is available to the
+       VMI layer.  This allows faults during the sequence to be properly
+       passed back to the guest kernel with the correct EIP.
+
+       Note that returning to userspace with interrupts disabled is an invalid
+       operation in a paravirtualized kernel, and the results of an attempt to
+       do so are undefined.
+
+       Also note that when issuing the VMI_IRET call, the userspace data
+       segments may have already been restored, so only the stack and code
+       segments can be assumed valid.
+
+       There is currently no support for IRET calls from a 16-bit stack
+       segment, which poses a problem for supporting certain userspace
+       applications which make use of high bits of ESP on a 16-bit stack.  How
+       to best resolve this is an open question.  One possibility is to
+       introduce a new VMI call which can operate on 16-bit segments, since it
+       is desirable to make the common case here as fast as possible.
+
+       Inputs:      ST(0) = New EIP
+                    ST(1) = New CS
+                    ST(2) = New Flags (including interrupt mask)
+                    ST(3) = New ESP (for userspace returns)
+                    ST(4) = New SS (for userspace returns)
+                    ST(5) = New ES (for v8086 returns)
+                    ST(6) = New DS (for v8086 returns)
+                    ST(7) = New FS (for v8086 returns)
+                    ST(8) = New GS (for v8086 returns)
+       Outputs:     None (does not return)
+       Clobbers:    None (does not return)
+       Segments:    CS / SS only
+
+    VMI_SYSEXIT
+
+       /* No C prototype provided */
+
+       For hypervisors and processors which support SYSENTER / SYSEXIT, the
+       VMI_SYSEXIT call is provided as a binary equivalent to the native
+       SYSENTER instruction.  Since interrupts must always be enabled in
+       userspace, the VMI version of this function always combines atomically
+       enabling interrupts with the return to userspace.
+
+       Inputs:      EDX = New EIP
+                    ECX = New ESP
+       Outputs:     None (does not return)
+       Clobbers:    None (does not return)
+       Segments:    CS / SS only
+
+
+   I/O CALLS
+
+    This set of calls incorporates I/O related calls - PIO, setting I/O
+    privilege level, and forcing memory writeback for device coherency.
+
+    VMI_INB
+    VMI_INW
+    VMI_INL
+
+       VMICALL VMI_UINT8  VMI_INB(VMI_UINT dummy, VMI_UINT port);
+       VMICALL VMI_UINT16 VMI_INW(VMI_UINT dummy, VMI_UINT port);
+       VMICALL VMI_UINT32 VMI_INL(VMI_UINT dummy, VMI_UINT port);
+
+       Input a byte, word, or doubleword from an I/O port.  These
+       instructions have binary equivalent semantics to native instructions.
+
+       Inputs:      EDX = port number
+                      EDX, rather than EAX is used, because the native
+                      encoding of the instruction may use this register
+                      implicitly.
+       Outputs:     EAX = port value
+       Clobbers:    Memory only
+       Segments:    Standard
+
+    VMI_OUTB
+    VMI_OUTW
+    VMI_OUTL
+
+       VMICALL void VMI_OUTB(VMI_UINT value, VMI_UINT port);
+       VMICALL void VMI_OUTW(VMI_UINT value, VMI_UINT port);
+       VMICALL void VMI_OUTL(VMI_UINT value, VMI_UINT port);
+
+       Output a byte, word, or doubleword to an I/O port.  These
+       instructions have binary equivalent semantics to native instructions.
+
+       Inputs:      EAX = port value
+                    EDX = port number
+       Outputs:     None
+       Clobbers:    None
+       Segments:    Standard
+
+    VMI_INSB
+    VMI_INSW
+    VMI_INSL
+
+       /* Not expressible as C functions */
+
+       Input a string of bytes, words, or doublewords from an I/O port.  These
+       instructions have binary equivalent semantics to native instructions.
+       They do not follow a C calling convention, and clobber only the same
+       registers as native instructions.
+
+       Inputs:      EDI = destination address
+                    EDX = port number
+                    ECX = count
+       Outputs:     None
+       Clobbers:    ESI, ECX, Memory
+       Segments:    Standard
+
+    VMI_OUTSB
+    VMI_OUTSW
+    VMI_OUTSL
+
+       /* Not expressible as C functions */
+
+       Output a string of bytes, words, or doublewords to an I/O port.  These
+       instructions have binary equivalent semantics to native instructions.
+       They do not follow a C calling convention, and clobber only the same
+       registers as native instructions.
+
+       Inputs:      ESI = source address
+                    EDX = port number
+                    ECX = count
+       Outputs:     None
+       Clobbers:    ESI, ECX
+       Segments:    Standard
+
+    VMI_IODelay
+
+       VMICALL void VMI_IODelay(void);
+
+       Delay the processor by time required to access a bus register.  This is
+       easily implemented on native hardware by an access to a bus scratch
+       register, but is typically not useful in a virtual machine.  It is
+       paravirtualized to remove the overhead implied by executing the native
+       delay.
+
+       Inputs:      None
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_SetIOPLMask
+
+       VMICALL void VMI_SetIOPLMask(VMI_UINT32 mask);
+
+       Set the IOPL mask of the processor to allow userspace to access I/O
+       ports.  Note the mask is pre-shifted, so an IOPL of 3 would be
+       expressed as (3 << 12).  If the guest chooses to use IOPL to allow
+       CPL-3 access to I/O ports, it must explicitly set and restore IOPL
+       using these calls; attempting to set the IOPL flags with popf or iret
+       may produce no result.
+
+       Inputs:      EAX = Mask
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_WBINVD
+
+       VMICALL void VMI_WBINVD(void);
+
+       Write back and invalidate the data cache.  This is used to synchronize
+       I/O memory.
+
+       Inputs:      None
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_INVD
+
+       This instruction is deprecated.  It is invalid to execute in a virtual
+       machine.  It is documented here only because it is still declared in
+       the interface, and dropping it required a version change.
+
+
+   APIC CALLS
+
+    APIC virtualization is currently quite simple.  These calls support the
+    functionality of the hardware APIC in a form that allows for more
+    efficient implementation in a hypervisor, by avoiding trapping access to
+    APIC memory.  The calls are kept simple to make the implementation
+    compatible with native hardware.  The APIC must be mapped at a page
+    boundary in the processor virtual address space.
+
+    VMI_APICWrite
+
+       VMICALL void VMI_APICWrite(void *reg, VMI_UINT32 value);
+
+       Write to a local APIC register.  Side effects are the same as native
+       hardware APICs.
+
+       Inputs:      EAX = APIC register address
+                    EDX = value to write
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_APICRead
+
+       VMICALL VMI_UINT32 VMI_APICRead(void *reg);
+
+       Read from a local APIC register.  Side effects are the same as native
+       hardware APICs.
+
+       Inputs:      EAX = APIC register address
+       Outputs:     EAX = APIC register value
+       Clobbers:    Standard
+       Segments:    Standard
+
+
+   TIMER CALLS
+
+    The VMI interfaces define a highly accurate and efficient timer interface
+    that is available when running inside of a hypervisor.  This is an
+    optional but highly recommended feature which avoids many of the problems
+    presented by classical timer virtualization.  It provides notions of
+    stolen time, counters, and wall clock time which allows the VM to
+    get the most accurate information in a way which is free of races and
+    legacy hardware dependence.
+
+    VMI_GetWallclockTime
+
+       VMI_NANOSECS VMICALL VMI_GetWallclockTime(void);
+
+       VMI_GetWallclockTime returns the current wallclock time as the number
+       of nanoseconds since the epoch.  Nanosecond resolution along with the
+       64-bit unsigned type provide over 580 years from epoch until rollover.
+       The wallclock time is relative to the host's wallclock time.
+
+       Inputs:      None
+       Outputs:     EAX = low word, wallclock time in nanoseconds
+                    EDX = high word, wallclock time in nanoseconds
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_WallclockUpdated
+
+       VMI_BOOL     VMICALL VMI_WallclockUpdated(void);
+
+       VMI_WallclockUpdated returns TRUE if the wallclock time has changed
+       relative to the real cycle counter since the previous time that
+       VMI_WallclockUpdated was polled.  For example, while a VM is suspended,
+       the real cycle counter will halt, but wallclock time will continue to
+       advance.  Upon resuming the VM, the first call to VMI_WallclockUpdated
+       will return TRUE.
+
+       Inputs:      None
+       Outputs:     EAX = 0 for FALSE, 1 for TRUE
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_GetCycleFrequency
+
+       VMICALL VMI_CYCLES VMI_GetCycleFrequency(void);
+
+       VMI_GetCycleFrequency returns the number of cycles in one second.  This
+       value can be used by the guest to convert between cycles and other time
+       units.
+
+       Inputs:      None
+       Outputs:     EAX = low word, cycle frequency
+                    EDX = high word, cycle frequency
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_GetCycleCounter
+
+       VMICALL VMI_CYCLES VMI_GetCycleCounter(VMI_UINT32 whichCounter);
+
+       VMI_GetCycleCounter returns the current value, in cycles units, of the
+       counter corresponding to 'whichCounter' if it is one of
+       VMI_CYCLES_REAL, VMI_CYCLES_AVAILABLE or VMI_CYCLES_STOLEN.
+       VMI_GetCycleCounter returns 0 for any other value of
'whichCounter'.
+
+       Inputs:      EAX = counter index, one of
+                        #define VMI_CYCLES_REAL        0
+                        #define VMI_CYCLES_AVAILABLE   1
+                        #define VMI_CYCLES_STOLEN      2
+       Outputs:     EAX = low word, cycle counter
+                    EDX = high word, cycle counter
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_SetAlarm
+
+       VMICALL void VMI_SetAlarm(VMI_UINT32 flags, VMI_CYCLES expiry,
+                                 VMI_CYCLES period);
+
+       VMI_SetAlarm is used to arm the vcpu's alarms.  The 'flags'
parameter
+       is used to specify which counter's alarm is being set
(VMI_CYCLES_REAL
+       or VMI_CYCLES_AVAILABLE), how to deliver the alarm to the vcpu
+       (VMI_ALARM_WIRED_IRQ0 or VMI_ALARM_WIRED_LVTT), and the mode
+       (VMI_ALARM_IS_ONESHOT or VMI_ALARM_IS_PERIODIC).  If the alarm is set
+       against the VMI_ALARM_STOLEN counter or an undefined counter number,
+       the call is a nop.  The 'expiry' parameter indicates the expiry
of the
+       alarm, and for periodic alarms, the 'period' parameter indicates
the
+       period of the alarm.  If the value of 'period' is zero, the
alarm is
+       armed as a one-shot alarm regardless of the mode specified by
'flags'.
+       Finally, a call to VMI_SetAlarm for an alarm that is already armed is
+       equivalent to first calling VMI_CancelAlarm and then calling
+       VMI_SetAlarm, except that the value returned by VMI_CancelAlarm is not
+       accessible.
+
+       /* The alarm interface 'flags' bits. [TBD: exact format of
'flags'] */
+
+       Inputs:      EAX   = flags value, cycle counter number or'ed with
+                        #define VMI_ALARM_WIRED_IRQ0   0x00000000
+                        #define VMI_ALARM_WIRED_LVTT   0x00010000
+                        #define VMI_ALARM_IS_ONESHOT   0x00000000
+                        #define VMI_ALARM_IS_PERIODIC  0x00000100
+                    EDX   = low word, alarm expiry
+                    ECX   = high word, alarm expiry
+                    ST(0) = low word, alarm expiry
+                    ST(1) = high word, alarm expiry
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+   VMI_CancelAlarm
+
+       VMICALL VMI_BOOL VMI_CancelAlarm(VMI_UINT32 flags);
+
+       VMI_CancelAlarm is used to disarm an alarm.  The 'flags'
parameter
+       indicates which alarm to cancel (VMI_CYCLES_REAL or
+       VMI_CYCLES_AVAILABLE).  The return value indicates whether or not the
+       cancel succeeded.  A return value of FALSE indicates that the alarm was
+       already disarmed either because a) the alarm was never set or b) it was
+       a one-shot alarm and has already fired (though perhaps not yet
+       delivered to the guest).  TRUE indicates that the alarm was armed and
+       either a) the alarm was one-shot and has not yet fired (and will no
+       longer fire until it is rearmed) or b) the alarm was periodic.
+
+       Inputs:      EAX = cycle counter number
+       Outputs:     EAX = 0 for FALSE, 1 for TRUE
+       Clobbers:    Standard
+       Segments:    Standard
+
+
+   MMU CALLS
+
+    The MMU plays a large role in paravirtualization due to the large
+    performance opportunities realized by gaining insight into the guest
+    machine's use of page tables.  These calls are designed to accommodate
the
+    existing MMU functionality in the guest OS while providing the hypervisor
+    with hints that can be used to optimize performance to a large degree.
+
+    VMI_SetLinearMapping
+
+       VMICALL void VMI_SetLinearMapping(int slot, VMI_UINT32 va,
+                                         VMI_UINT32 pages, VMI_UINT32 ppn);
+
+       /* The number of VMI address translation slot */
+       #define VMI_LINEAR_MAP_SLOTS    4
+
+       Register a virtual to physical translation of virtual address range to
+       physical pages.  This may be used to register single pages or to
+       register large ranges.  There is an upper limit on the number of active
+       mappings, which should be sufficient to allow the hypervisor and VMI
+       layer to perform page translation without requiring dynamic storage.
+       Translations are only required to be registered for addresses used to
+       access page table entries through the VMI page table access functions.
+       The guest is free to use the provided linear map slots in a manner that
+       it finds most convenient.  Kernels which linearly map a large chunk of
+       physical memory and use page tables in this linear region will only
+       need to register one such region after initialization of the VMI.
+       Hypervisors which do not require linear to physical conversion hints
+       are free to leave these calls as NOPs, which is the default when
+       inlined into the native kernel.
+
+       Inputs:      EAX   = linear map slot
+                    EDX   = virtual address start of mapping
+                    ECX   = number of pages in mapping
+                    ST(0) = physical frame number to which pages are mapped
+       Outputs:     None
+       Clobbers:    Standard
+       Segments:    Standard
+
+    VMI_FlushTLB
+
+       VMICALL void VMI_FlushTLB(int how);
+
+       Flush all non-global mappings in the TLB, optionally flushing global
+       mappings as well.  The VMI_FLUSH_TLB flag should always be specified,
+       optionally or'ed with the VMI_FLUSH_GLOBAL flag.
+
+       Inputs:      EAX = flush type
+                       #define VMI_FLUSH_TLB            0x01
+                       #define VMI_FLUSH_GLOBAL         0x02
+       Outputs:     None
+       Clobbers:    Standard, memory (implied)
+       Segments:    Standard
+
+    VMI_InvalPage
+
+       VMICALL void VMI_InvalPage(VMI_UINT32 va);
+
+       Invalidate the TLB mapping for a single page or large page at the
+       given virtual address.
+
+       Inputs:      EAX = virtual address
+       Outputs:     None
+       Clobbers:    Standard, memory (implied)
+       Segments:    Standard
+
+   The remaining documentation here needs updating when the PTE accessors are
+   simplified.
+
+    70) VMI_SetPte
+
+        void VMI_SetPte(VMI_PTE pte, VMI_PTE *ptep);
+
+        Assigns a new value to a page table / directory entry. It is a
+        requirement that ptep points to a page that has already been
+        registered with the hypervisor as a page of the appropriate type
+	using the VMI_RegisterPageUsage function.
+
+    71) VMI_SwapPte
+
+        VMI_PTE VMI_SwapPte(VMI_PTE pte, VMI_PTE *ptep);
+
+        Write 'pte' into the page table entry pointed by
'ptep', and returns
+        the old value in 'ptep'.  This function acts atomically on the
PTE
+        to provide up to date A/D bit information in the returned value.
+
+    72) VMI_TestAndSetPteBit
+
+        VMI_BOOL VMI_TestAndSetPteBit(VMI_INT bit, VMI_PTE *ptep);
+
+        Atomically set a bit in a page table entry.  Returns zero if the bit
+        was not set, and non-zero if the bit was set.
+
+    73) VMI_TestAndClearPteBit
+
+        VMI_BOOL VMI_TestAndSetClearBit(VMI_INT bit, VMI_PTE *ptep);
+
+        Atomically clear a bit in a page table entry.  Returns zero if the bit
+        was not set, and non-zero if the bit was set.
+
+    74) VMI_SetPteLong
+    75) VMI_SwapPteLong
+    76) VMI_TestAndSetPteBitLong
+    77) VMI_TestAndClearPteBitLong
+
+        void VMI_SetPteLong(VMI_PAE_PTE pte, VMI_PAE_PTE *ptep);
+        VMI_PAE_PTE VMI_SwapPteLong(VMI_UINT64 pte, VMI_PAE_PTE *ptep);
+        VMI_BOOL VMI_TestAndSetPteBitLong(VMI_INT bit, VMI_PAE_PTE *ptep);
+        VMI_BOOL VMI_TestAndSetClearBitLong(VMI_INT bit, VMI_PAE_PTE *ptep);
+
+        These functions act identically to the 32-bit PTE update functions,
+        but provide support for PAE mode.  The calls are guaranteed to never
+        create a temporarily invalid but present page mapping that could be
+        accidentally prefetched by another processor, and all returned bits
+        are guaranteed to be atomically up to date.
+
+        One special exception is the VMI_SwapPteLong function only provides
+        synchronization against A/D bits from other processors, not against
+        other invocations of VMI_SwapPteLong.
+
+    78) VMI_ClonePageTable
+        VMI_ClonePageDirectory
+
+        #define VMI_MKCLONE(start, count) (((start) << 16) | (count))
+
+        void VMI_ClonePageTable(VMI_UINT32 dstPPN, VMI_UINT32 srcPPN,
+                                VMI_UINT32 flags);
+        void VMI_ClonePageDirectory(VMI_UINT32 dstPPN, VMI_UINT32 srcPPN,
+                                VMI_UINT32 flags);
+
+        These functions tell the hypervisor to allocate a page shadow
+        at the PT or PD level using a shadow template.  Because of the
+        availability of bits in the flags, these calls may be merged
+        together as well as flag the PAE-ness of the shadows.
+
+    80) VMI_RegisterPageUsage
+    81) VMI_ReleasePage
+
+        #define VMI_PAGE_PT              0x01
+        #define VMI_PAGE_PD              0x02
+        #define VMI_PAGE_PDP             0x04
+        #define VMI_PAGE_PML4            0x08
+        #define VMI_PAGE_GDT             0x10
+        #define VMI_PAGE_LDT             0x20
+        #define VMI_PAGE_IDT             0x40
+        #define VMI_PAGE_TSS             0x80
+
+        void  VMI_RegisterPageUsage(VMI_UINT32 ppn, int flags);
+        void  VMI_ReleasePage(VMI_UINT32 ppn, int flags);
+
+        These are used to register a page with the hypervisor as being of a
+        particular type, for instance, VMI_PAGE_PT says it is a page table
+        page.
+
+    85) VMI_SetDeferredMode
+
+        void VMI_SetDeferredMode(VMI_UINT32 deferBits);
+
+        Set the lazy state update mode to the specified set of bits.  This
+        allows the processor, hypervisor, or VMI layer to lazily update
+        certain CPU and MMU state.  When setting this to a more permissive
+        setting, no flush is implied, but when clearing bits in the current
+        defer mask, all pending state will be flushed.
+
+        The 'deferBits' is a mask specifying how to flush.
+
+            #define VMI_DEFER_NONE          0x00
+
+        Disallow all asynchronous state updates.  This is the default
+        state.
+
+            #define VMI_DEFER_MMU           0x01
+
+	Flush all pending page table updates.  Note that page faults,
+        invalidations and TLB flushes will implicitly flush all pending
+        updates.
+
+            #define VMI_DEFER_CPU           0x02
+
+        Allow CPU state updates to control registers to be deferred, with
+        the exception of updates that change FPU state.  This is useful
+        for combining a reload of the page table base in CR3 with other
+        updates, such as the current kernel stack.
+
+            #define VMI_DEFER_DT            0x04
+
+        Allow descriptor table updates to be delayed.  This allows the
+        VMI_UpdateGDT / IDT / LDT calls to be asynchronously queued.
+
+    86) VMI_FlushDeferredCalls
+
+        void VMI_FlushDeferredCalls(void);
+
+        Flush all asynchronous state updates which may be queued as
+        a result of setting deferred update mode.
+
+
+Appendix B - VMI C prototypes
+
+   Most of the VMI calls are properly callable C functions.  Note that for the
+   absolute best performance, assembly calls are preferable in some cases, as
+   they do not imply all of the side effects of a C function call, such as
+   register clobber and memory access.  Nevertheless, these wrappers serve as
+   a useful interface definition for higher level languages.
+
+   In some cases, a dummy variable is passed as an unused input to force
+   proper alignment of the remaining register values.
+
+   The call convention for these is defined to be standard GCC convention with
+   register passing.  The regparm call interface is documented at:
+
+   http://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html
+
+   Types used by these calls:
+
+   VMI_UINT64   64 bit unsigned integer
+   VMI_UINT32   32 bit unsigned integer
+   VMI_UINT16   16 bit unsigned integer
+   VMI_UINT8    8 bit unsigned integer
+   VMI_INT      32 bit integer
+   VMI_UINT     32 bit unsigned integer
+   VMI_DTR      6 byte compressed descriptor table limit/base
+   VMI_PTE      4 byte page table entry (or page directory)
+   VMI_LONG_PTE 8 byte page table entry (or PDE or PDPE)
+   VMI_SELECTOR 16 bit segment selector
+   VMI_BOOL     32 bit unsigned integer
+   VMI_CYCLES   64 bit unsigned integer
+   VMI_NANOSECS 64 bit unsigned integer
+
+
+   #ifndef VMI_PROTOTYPES_H
+   #define VMI_PROTOTYPES_H
+
+   /* Insert local type definitions here */
+   typedef struct VMI_DTR {
+      uint16 limit;
+      uint32 offset __attribute__ ((packed));
+   } VMI_DTR;
+
+   typedef struct APState {
+      VMI_UINT32 cr0;
+      VMI_UINT32 cr2;
+      VMI_UINT32 cr3;
+      VMI_UINT32 cr4;
+
+      VMI_UINT64 efer;
+
+      VMI_UINT32 eip;
+      VMI_UINT32 eflags;
+      VMI_UINT32 eax;
+      VMI_UINT32 ebx;
+      VMI_UINT32 ecx;
+      VMI_UINT32 edx;
+      VMI_UINT32 esp;
+      VMI_UINT32 ebp;
+      VMI_UINT32 esi;
+      VMI_UINT32 edi;
+      VMI_UINT16 cs;
+      VMI_UINT16 ss;
+
+      VMI_UINT16 ds;
+      VMI_UINT16 es;
+      VMI_UINT16 fs;
+      VMI_UINT16 gs;
+      VMI_UINT16 ldtr;
+
+      VMI_UINT16 gdtrLimit;
+      VMI_UINT32 gdtrBase;
+      VMI_UINT32 idtrBase;
+      VMI_UINT16 idtrLimit;
+   } APState;
+
+   #define VMICALL __attribute__((regparm(3)))
+
+   /* CORE INTERFACE CALLS */
+   VMICALL void VMI_Init(void);
+
+   /* PROCESSOR STATE CALLS */
+   VMICALL void     VMI_DisableInterrupts(void);
+   VMICALL void     VMI_EnableInterrupts(void);
+
+   VMICALL VMI_UINT VMI_GetInterruptMask(void);
+   VMICALL void     VMI_SetInterruptMask(VMI_UINT mask);
+
+   VMICALL void     VMI_Pause(void);
+   VMICALL void     VMI_Halt(void);
+   VMICALL void     VMI_Shutdown(void);
+   VMICALL void     VMI_Reboot(VMI_INT how);
+
+   #define VMI_REBOOT_SOFT 0x0
+   #define VMI_REBOOT_HARD 0x1
+
+   void VMI_SetInitialAPState(APState *apState, VMI_UINT32 apicID);
+
+   /* DESCRIPTOR RELATED CALLS */
+   VMICALL void         VMI_SetGDT(VMI_DTR *gdtr);
+   VMICALL void         VMI_SetIDT(VMI_DTR *idtr);
+   VMICALL void         VMI_SetLDT(VMI_SELECTOR ldtSel);
+   VMICALL void         VMI_SetTR(VMI_SELECTOR ldtSel);
+
+   VMICALL void         VMI_GetGDT(VMI_DTR *gdtr);
+   VMICALL void         VMI_GetIDT(VMI_DTR *idtr);
+   VMICALL VMI_SELECTOR VMI_GetLDT(void);
+   VMICALL VMI_SELECTOR VMI_GetTR(void);
+
+   VMICALL void         VMI_WriteGDTEntry(void *gdt,
+                                          VMI_UINT entry,
+                                          VMI_UINT32 descLo,
+                                          VMI_UINT32 descHi);
+   VMICALL void         VMI_WriteLDTEntry(void *gdt,
+                                          VMI_UINT entry,
+                                          VMI_UINT32 descLo,
+                                          VMI_UINT32 descHi);
+   VMICALL void         VMI_WriteIDTEntry(void *gdt,
+                                          VMI_UINT entry,
+                                          VMI_UINT32 descLo,
+                                          VMI_UINT32 descHi);
+
+   /* CPU CONTROL CALLS */
+   VMICALL void       VMI_WRMSR(VMI_UINT64 val, VMI_UINT32 reg);
+   VMICALL void       VMI_WRMSR_SPLIT(VMI_UINT32 valLo, VMI_UINT32 valHi,
+                                      VMI_UINT32 reg);
+
+   /* Not truly a proper C function; use dummy to align reg in ECX */
+   VMICALL VMI_UINT64 VMI_RDMSR(VMI_UINT64 dummy, VMI_UINT32 reg);
+
+   VMICALL void VMI_SetCR0(VMI_UINT val);
+   VMICALL void VMI_SetCR2(VMI_UINT val);
+   VMICALL void VMI_SetCR3(VMI_UINT val);
+   VMICALL void VMI_SetCR4(VMI_UINT val);
+
+   VMICALL VMI_UINT32 VMI_GetCR0(void);
+   VMICALL VMI_UINT32 VMI_GetCR2(void);
+   VMICALL VMI_UINT32 VMI_GetCR3(void);
+   VMICALL VMI_UINT32 VMI_GetCR4(void);
+
+   VMICALL void       VMI_CLTS(void);
+
+   VMICALL void       VMI_SetDR(VMI_UINT32 num, VMI_UINT32 val);
+   VMICALL VMI_UINT32 VMI_GetDR(VMI_UINT32 num);
+
+   /* PROCESSOR INFORMATION CALLS */
+
+   VMICALL VMI_UINT64 VMI_RDTSC(void);
+   VMICALL VMI_UINT64 VMI_RDPMC(VMI_UINT64 dummy, VMI_UINT32 counter);
+
+   /* STACK / PRIVILEGE TRANSITION CALLS */
+   VMICALL void VMI_UpdateKernelStack(void *tss, VMI_UINT32 esp0);
+
+   /* I/O CALLS */
+   /* Native port in EDX - use dummy */
+   VMICALL VMI_UINT8  VMI_INB(VMI_UINT dummy, VMI_UINT port);
+   VMICALL VMI_UINT16 VMI_INW(VMI_UINT dummy, VMI_UINT port);
+   VMICALL VMI_UINT32 VMI_INL(VMI_UINT dummy, VMI_UINT port);
+
+   VMICALL void VMI_OUTB(VMI_UINT value, VMI_UINT port);
+   VMICALL void VMI_OUTW(VMI_UINT value, VMI_UINT port);
+   VMICALL void VMI_OUTL(VMI_UINT value, VMI_UINT port);
+
+   VMICALL void VMI_IODelay(void);
+   VMICALL void VMI_WBINVD(void);
+   VMICALL void VMI_SetIOPLMask(VMI_UINT32 mask);
+
+   /* APIC CALLS */
+   VMICALL void       VMI_APICWrite(void *reg, VMI_UINT32 value);
+   VMICALL VMI_UINT32 VMI_APICRead(void *reg);
+
+   /* TIMER CALLS */
+   VMICALL VMI_NANOSECS VMI_GetWallclockTime(void);
+   VMICALL VMI_BOOL     VMI_WallclockUpdated(void);
+
+   /* Predefined rate of the wallclock. */
+   #define VMI_WALLCLOCK_HZ       1000000000
+
+   VMICALL VMI_CYCLES VMI_GetCycleFrequency(void);
+   VMICALL VMI_CYCLES VMI_GetCycleCounter(VMI_UINT32 whichCounter);
+
+   /* Defined cycle counters */
+   #define VMI_CYCLES_REAL        0
+   #define VMI_CYCLES_AVAILABLE   1
+   #define VMI_CYCLES_STOLEN      2
+
+   VMICALL void     VMI_SetAlarm(VMI_UINT32 flags, VMI_CYCLES expiry,
+                                 VMI_CYCLES period);
+   VMICALL VMI_BOOL VMI_CancelAlarm(VMI_UINT32 flags);
+
+   /* The alarm interface 'flags' bits. [TBD: exact format of
'flags'] */
+   #define VMI_ALARM_COUNTER_MASK 0x000000ff
+
+   #define VMI_ALARM_WIRED_IRQ0   0x00000000
+   #define VMI_ALARM_WIRED_LVTT   0x00010000
+
+   #define VMI_ALARM_IS_ONESHOT   0x00000000
+   #define VMI_ALARM_IS_PERIODIC  0x00000100
+
+   /* MMU CALLS */
+   VMICALL void VMI_SetLinearMapping(int slot, VMI_UINT32 va,
+                                     VMI_UINT32 pages, VMI_UINT32 ppn);
+
+   /* The number of VMI address translation slot */
+   #define VMI_LINEAR_MAP_SLOTS    4
+
+   VMICALL void VMI_InvalPage(VMI_UINT32 va);
+   VMICALL void VMI_FlushTLB(int how);
+
+   /* Flags used by VMI_FlushTLB call */
+   #define VMI_FLUSH_TLB            0x01
+   #define VMI_FLUSH_GLOBAL         0x02
+
+   #endif
+
+
+Appendix C - Sensitive x86 instructions in the paravirtual environment
+
+  This is a list of x86 instructions which may operate in a different manner
+  when run inside of a paravirtual environment.
+
+	ARPL - continues to function as normal, but kernel segment registers
+	       may be different, so parameters to this instruction may need
+	       to be modified. (System)
+
+	IRET - the IRET instruction will be unable to change the IOPL, VM,
+	       VIF, VIP, or IF fields. (System)
+
+	       the IRET instruction may #GP if the return CS/SS RPL are
+	       below the CPL, or are not equal. (System)
+
+	LAR  - the LAR instruction will reveal changes to the DPL field of
+	       descriptors in the GDT and LDT tables. (System, User)
+
+	LSL  - the LSL instruction will reveal changes to the segment limit
+	       of descriptors in the GDT and LDT tables. (System, User)
+
+	LSS  - the LSS instruction may #GP if the RPL is not set properly.
+	       (System)
+
+	MOV  - the mov %seg, %reg instruction may reveal a different RPL
+	       on the segment register. (System)
+
+	       The mov %reg, %ss instruction may #GP if the RPL is not set
+	       to the current CPL. (System)
+
+	POP  - the pop %ss instruction may #GP if the RPL is not set to
+	       the appropriate CPL. (System)
+
+	POPF - the POPF instruction will be unable to set the hardware
+	       interrupt flag. (System)
+
+	PUSH - the push %seg instruction may reveal a different RPL on the
+	       segment register. (System)
+
+	PUSHF- the PUSHF instruction will reveal a possible different IOPL,
+	       and the value of the hardware interrupt flag, which is always
+	       set.  (System, User)
+
+	SGDT - the SGDT instruction will reveal the location and length of
+	       the GDT shadow instead of the guest GDT. (System, User)
+
+	SIDT - the SIDT instruction will reveal the location and length of
+	       the IDT shadow instead of the guest IDT. (System, User)
+
+	SLDT - the SLDT instruction will reveal the selector used for
+	       the shadow LDT rather than the selector loaded by the guest.
+	       (System, User).
+
+	STR  - the STR instruction will reveal the selector used for the
+	       shadow TSS rather than the selector loaded by the guest.
+	       (System, User).