Folks,
Please find attached a proposal for the Xen development roadmap for the
next 18 months or so.
The roadmap gives a "state of the nation" summary and provides
suggestions as to priorities going forward. I''ve tried to be inclusive
and collect together many of the various features that folks have asked
for, along with some implementation ideas to guide developers. The
document ends with a table that has a suggested timeline for landing
some of the more critical features. [NB: The doc was written over the
period of a couple of months, so a few of the work items mentioned are
already now complete and checked-in.]
Lets discuss the proposal on xen-devel and at the xen mini summit, then
get it tabulated onto the wiki as a live document.
Best,
Ian
[latex text version follows, PDF attached]
\title{The Xen Roadmap}
\author{Ian Pratt, July 2006}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\maketitle
Since around October 2005 the main focus of the Xen project has really
been around stability: getting the 3.0.0 release out, and then ongoing
hardening in preparation for FC5 and SLES10. We''re now in the
fortunate position of having really very few outstanding bugs, and now
have excellent automated test infrastructure to help us maintain this
level of quality.
It''s now time to turn thoughts to further development and work out
where we want to take the project toward the next major release, Xen
3.1 (or 4.0, whatever we end up calling it). This document lists what
I believe are the priorities over the coming months, and hence what
the core Xen team will be investing effort in.
\section{Releases}
%%%%%%%%%%%%%%%%%%
Rather than the `big bang'' 12 month development cycle used in Xen 1.0,
2.0 and 3.0, we want to make the development toward the next major
release far more incremental. We aim to continue the current practise
of going through a stabilization phase and having point releases every
10-12 weeks. Just as with Linux, the point release will be maintained
with bug fixes until the next point release.
One side effect of this approach is that at the time of a release not
all features that are new in the code base may necessarily be stable
in time for the release. We won''t hold the release, but will simply
disable such features or document the issues. The aim is obviously to
avoid regressions in functionality that worked in earlier releases.
\section{Performance, Scalability}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
During the Xen 3 development cycle the vast majority of effort went in
to correctness rather than performance and scalability. It''s pretty
clear from looking at historic benchmark data that some performance
bugs have crept in along the way: there really is no good reason why
xen3 should perform worse than xen2 on any benchmark, but measurements
indicate there are currently regressions on some benchmarks. Clearly,
investigation is called for. I expect there to be quite a bit of low
hanging fruit, in particular it is likely that various of the Linux
kernel version upgrades have broken certain assumptions causing us to
exercise `slow paths'' rather than the fast path we expect it to be
using. These need to be tracked down and fixed.
We''re actually quite well-armed with performance monitoring tools these
days, so hopefully this shouldn''t be too difficult: xen-oprofile is a
sample-based profile system useful for looking at system-wide CPU
consumption. The xen software performance counters are useful for
tracking event occurrences in Xen (and hence spotting anomalously high
counts of supposedly rare events), and can also be used to collect
various histograms of scheduler and memory management data. There is
also xentrace which can be enabled to collect timestamped trace
records into a buffer to enable detailed event time-lines to be
collected.
We expect the vast majority of Xen deployments to be on two socket
server boxes, with some four socket, and we expect dual core to be
commonplace. Hence, we believe that the sweet spot to optimize xen for
is 1-8 CPU systems. 32 way and larger systems are currently supported
and getting good performance on them is certainly desirable in the
mid-term. However, we must endeavour to ensure that such optimizations
do not harm smaller system performance. If necessary, compile time
target selection could be used, but I hope this won''t be required. One
of the key features which will help both small and large system
performance is support for NUMA-aware memory allocation. This is
useful even on two socket AMD boxes. I expect to see the core Xen
mechanisms in the tree in short order, but implementing page migration
(particularly the policy for doing so) will be a longer term goal.
NUMA topology-aware CPU scheduling will also need to be developed, and
is discussed in a later section.
\section{Guest API stability}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The Xen 3 Guest API is intended to be a stable interface that will be
maintained in a backward compatible fashion. Old 3.0 guests should run
on a new hypervisor. The converse of running new guests on an old
hypervisor has not been a commitment, but we should obviously plan to
transition toward this. This becomes more important as soon as a given
version of Xen starts gaining wide use in an Enterprise Linux distro
(e.g. SLES10).
The xen guest API includes all the virtual IO interfaces
(e.g. netfront, blkfront etc). We expect there to be some evolution
in these protocols, but nothing that can''t be supported in a
compatible fashion: xenbus provides the necessary mechanisms for
enabling feature selection, or even selection of alternate front or
backend drivers.
So far, there has been no commitment to maintain a stable interface
between privileged domains (dom0) and xen, or between xend and
dom0. We currently expect xen, the dom0 kernel, and the xend tool
stack to be a ``matched set''''. These interfaces will continue
to evolve
for at least the next six months, so no stability is guaranteed,
though interface breakage will be avoided where possible as it''s
clearly inconvenient for developers. Interface version numbers should be
updated when this happens to avoid subtle incompatibilities.
The continued evolution of the privileged domain interfaces is being
driven by a number of factors: changes such as the new pci device
pass-through code, support for IOMMUs, integration of ia64 and ppc,
more fine-grained security capability delegation. It should be a long
term goal to stabilize this interface, but for the moment 2.6 Linux is
in the favoured position of being the in-tree privileged domain OS,
with NetBSD and Solaris playing catchup.
One of the key APIs we expect to evolve and then stabilize quickly in
the next 2-3 months is the xen control API, encompassing all the
various XML parameters that configure and control guest domains. The
roadmap for the xen control stack is set out later in this document.
\section{OS Support}
%%%%%%%%%%%%%%%%%%%%
The following OS kernels have been ported to the Xen3 32b guest ABI:
Linux 2.6.16, 2.6.9-34.EL, 2.4.21-40.EL, 2.6.5-7.252, NetBSD3,
FreeBSD7.0 and OpenSolaris10. Work is planned to update the Plan9 port
to xen 3, and the API stability will hopefully encourage other OSes to
be ported too, such as OpenBSD. The only x86\_64 OS port is currently
2.6.16, but other ports are underway.
Since the 3.0.0 release there have been various backward compatible
enhancements to the Xen guest API, such as the `feature flags'' and
`transfer page''. These mechanisms are intended to make backward
compatibility easier to maintain in future, and will be used to enable
features such as supporting 32b guests on a 64b hypervisor, running
xen kernels on bare metal, and running xen kernels as HVM (fully
virtualized) guests. We would urge maintainers of kernel ports to
adopt the new feature flags and transfer page ASAP if they haven''t
already done so.
\section{Getting xen support in kernel.org Linux}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
The work to get xen support into upstream kernel.org Linux is a
crucial work item for the xen community. Although it costs
considerable man effort right now, it should reduce the maintenance
burden in the long run as hypervisor support will be a more visible
consideration for Linux developers, and hopefully treated as a first
class citizen.
The full xen support for Linux patch maintained in the main xen
repository is really quite large. As well as supporting Linux as an
optimized SMP guest, it has all the functionality necessary to run
Linux as a domain 0 with access to physical hardware, support for IO
virtualization backends, support for the control tools etc.
It is clear that getting such a large patch into kernel.org in one go
is infeasible, so a more incremental approach has been adopted. Our
initial aim is to submit patches that enable Linux to run as a simple
Xen guest, without various of the more invasive paravirtualization
optimizations that provide rather better virtual memory and SMP
performance. A patchset has been prepared by Christian Limpach,
Chris Wright and Jeremy Fitzhardinge, and is being iterated on LKML.
The path into kernel.org has been muddied by the discussion around
VMware''s VMI proposal. VMI proposes an abstraction layer for the
hypervisor to guest API used for CPU and memory management
virtualization, and doesn''t attempt to address the virtual IO,
hardware access, and control tool APIs that the Xen patches
address. VMI is also currently only addressing 32b x86 (Xen is
currently x86\_64 and ia64 too).
Comparing VMI with a subset of the xen patch, there''s actually quite a
lot in common. The bulk of both patches refactor i386 code to provide
hooks, and it is hoped that these common changes can be upstreamed
quickly, reducing the size of the external patch that must be
maintained. Rusty Russell is taking the lead on this. At some point
there will be a discussion about what the correct hypervisor API
abstraction is, and hence the extent to which paravirtualization can
be exploited to improve performance. The Xen team will be arguing
vigorously that a rich interface that is part of the kernel source
(and hence GPL) is more easily maintainable, and enables `deeper''
paravirtualization, and hence more opportunities for performance
enhancements.
It is conceivable that what ends up being merged into kernel.org may
require some changes to the xen 3 guest ABI. We would certainly hope
that these could be made backwards compatible with existing xen guest
OSes, but we may have no choice other than to rev to a Xen 4 ABI. This
would be a great shame as ABI stability is a key hypervisor
requirement, but this would at least be a one off change.
\section{Control tool stack}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Following on from the discussions at the last Xen summit, a number of
significant changes are planned for the xen control tool stack over
the next few months. In the mid-term, the DMTF CIM model for VM
life-cycle management from the virtualization and partitioning working
group is likely to emerge as the standard for configuring and managing
VMs. Many of the folks in the Xen community are already working in the
DMTF in support of the CIM model, as well as Microsoft in their
Carmine tools. The joint IBM/Novell/XenSource project to create CIM
providers for Xen is of great strategic importance. However,
management via CIM is quite heavyweight and intricate for some
scenarios, so also having a simpler management API makes
sense. Indeed, having such an API will be useful for building the CIM
providers themselves.
{\em XML config file and conversion tools.} The first stage of
xend development will be to switch to using XML for all configuration
data. We are in the process of drafting a specification for an XML
data-model for storing all VM configuration data, and will be
circulating this on xen-devel/xen-cim shortly. This scheme is
`inspired by'' the CIM data model, but the hierarchy is somewhat
flattened and simplified to reflect xen''s requirements and provide an
easy to navigate model. The intention is that XML config files
conforming to this schema would replace the current python and SXP
xend config files. Since this is a user-visible change, creation of
migration tools will be required. Config files would no longer live
under /etc, but would be loaded into xend when a VM is created, and
then stored as plain XML files under /var.
{\em xend VM life cycle management and storage extensions.} Another
addition will be implementation of some simple VM life-cycle
management inside xend. The current tools already have a very limited
form of this implemented by the xendomains script which will preserve
VMs across host reboots using save/restore. Adding VM life-cycle
management to xend means that we will store state for VMs even if they
are not currently running. This will enable us to make operations like
VM save/restore less dangerous from a user point of view by tracking
the resources (in particular, disk images) they have reserved even
when they are suspended. Further, support for simple storage
management will be added to xend, enabling call-outs to create new
disk images, create CoW snapshots etc. These will be implemented in
scripts that provide the functionality for different storage backends
e.g. LVM, file based, qcow, etc.
In previous and current Xen releases the protocol between `xm'' and
`xend'' has not been documented and has undergone rapid
change. Although the internal xmlib API has generally remained
relatively stable this has not been a popular interface for developers
building on top of Xen -- most have resorted to using `expect'' scripts
to drive the xm command line.
{\em XML-RPC xen control API plus C/C++/perl/python bindings.} It''s
now important that we define a suitable message protocol and
associated API, rapidly switch xend and xm over to using it, and then
provide client bindings for common languages
e.g. C++/perl/python. Libvirt will hopefully help fulfil this
role. Draft specifications for the message protocol are being prepared
and will be circulated shortly. The scheme is xml-rpc based, using an
SSL secured https transport to enable secure remote management, or
unix domain sockets for lightweight local management.
The protocol will support a notion of logging in as a given user,
using PAM to authenticate on the server. Since some of the RPC
commands may be long running (e.g. a VM relocation), the protocol
supports
the notion of tasks running asynchronously in the background. The
client can poll for their completion or list outstanding tasks.
In addition to securing the control protocol, we also need to secure
the networking connections used for VM relocation. The intention is to
split the VM relocation operation into two, issuing a relocate\_receive
that generates a token that then must be presented when initiating a
relocate\_send. Since these network connections are performance
critical, it is important we retain the option to have just
authentication without mandating encryption.
A further area where work is clearly required is developing a decent
web GUI for Xen. There have been a number of previous attempts at this
by various folk, but none have really gained traction. We would
envisage that the GUI would be implemented using presentation layer
code that would communicate with xend via the control protocol and
generate HTML/javascript to return to the browser. Possibly picking
one of the existing web UI''s and trying to get it in-tree will help
focus efforts.
\section{Virtual Hard Disk Images}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Xen 3.0 supports a wide range of storage options for guest virtual
disks, including physical partitions (LUNs), LVM volumes, and `loop''
files. Use of loop files has proved popular as they support sparse
allocation, and are very easy from a management/backup point of view
(e.g. you can just copy the files around). However, the loop driver
has some serious deficiencies, in particular it buffers writes very
aggressively, which can affect guest filesystem correctness in the
event of a host crash, and can even cause out-of-memory kernel crashes
in domain0 under heavy write load. Further, using sparse files
requires care to ensure the sparseness is preserved when copying, and
there is no header in which metadata relating back to the VM can be
stored.
Given the popularity of the file-backed model, providing a robust and
high performance solution that supports it is highly
desirable. Rather than using raw image files, it makes sense to move
to a format that supports header metadata, sparse allocation
independent of the underlying filesystem, and copy-on-write
support. We have spent time evaluating a number of existing formats:
VMware VMDK, Microsoft VHD, and QEMU QCOW. VMDK is a hodgepodge union
of several different formats used by VMware, and doesn''t have much to
recommend it. Further, the licencing terms aren''t entirely clear about
its GPL compatibility. Microsoft''s VHD format is pretty nice, but
explicitly doesn''t allow open source implementations. QEMU QCOW looks
to be the best of the bunch: It''s been part of the QEMU stable for
over 18 months and had quite widespread use, and supports advanced
options like compression and AES encryption as well as sparse
allocation and copy-on-write.
The world doesn''t need another virtual hard disk format, so with
Fabrice Bellard''s blessing (QEMU author) we''re strongly
advocating
that the Xen project adopt it. The licence on the current QCOW
implementation is BSD, which makes implementation with Xen and even 3rd
party closed-source tools easy (e.g. Virtual-to-Physical transfer
utilities).
Given the various problems with the `loop'' driver, this
doesn''t seem a
good starting point for implementing qcow support. The easiest
approach seems to be to build on the `blktap'' approach that is already
in the tree and provides a way of implementing virtual block devices
in user space. Work is well under way to implement a `ublkback'' driver
that supports all of the various qemu file format plugins. A special
high-performance qcow plugin is also under development, that supports
better metadata caching, asynchronous IO, and allows request
reordering with appropriate safety barriers to enforce correctness. It
remains both forward and backward compatible with existing qcow disk
images, but makes adjustments to qemu''s default allocation policy when
creating new disks such as to optimize performance.
\section{Resource Control}
%%%%%%%%%%%%%%%%%%%%%%%%%%
One of the big deficiencies with Xen 3.0.2 is the static assignment of
VCPUs to single physical CPUs. Although domain creation attempts to do
some crude load-balancing placement on physical CPUs the assignment
is not updated, which can lead to gross imbalance between CPUs if
e.g. all even numbered VMs were to exit. Currently manual intervention
with `xm vcpu-pin'' is required to rebalance things.
A new CPU scheduler has recently been completed which should solve these
issues. It supports automatic migration of VCPUs between their
allowable CPU set, and actively balances VCPUs across CPUs
in an attempt to maximise throughput.
The scheduler supports a notion of `weighted fair share'', enabling the
relative weights of guests to be set e.g. this guest should be able to
get twice as much CPU as this guest (assuming they are both CPU bound
and runnable). Further percentage CPU ceilings may be set to constrain
the
consumption of a guest, e.g. limiting it to 10\% of a CPU even if the
CPU is otherwise idle. [This non-work conserving option is useful in
hosting environments where it is sometimes desirable to stop customers
having the opportunity to get used to more resource than they''re
paying for.]
For multi VCPU guests the scheduler tries to ensure that each
(runnable) VCPU accrues CPU time in roughly equal fashion. This avoids
bad interactions with the guest OSes internal scheduler. Currently, no
attempt is made to gang schedule VCPUs belonging to the same
guest. For most workloads this seems to work OK. In future we may have
to investigate schemes that switch to gang scheduling under certain
circumstances, or otherwise use bad pre-emption avoidance (e.g. don''t
pre-empt while kernel locks held) or bad pre-emption mitigation
strategies (e.g. directed yield). It may be possible to dynamically
spot groups of VCPUs that are actively communicating and gang schedule
them. We will need to extend the scheduler to better understand CPU
topology and make informed scheduling decisions in the presence of
hyperthreading or NUMA architectures (in the short term, we can use
CPU affinity settings to assist this). Due to potential information
leakage through cache timing attacks between domains sharing the same
hyperthreaded CPU core, we may need to ensure that hyperthreads
belonging to the same core are gang scheduled for the same domain (of
shared with a trusted IO domain).
One other scheduling feature that has been discussed is instrumenting
domain 0 to enable the work that it performs on behalf of other guests
to be accounted to those guests. The simplest way to do this is just
to do some simple counting of grant transfer and grant map events for
each domain, and then bill the CPU time consumed by domain 0 in
proportion (with an adjustment factor to account for the CPU
differences between disk and network IO). Although simple, this scheme
would probably yield most of the benefit, and avoid CPU-bound domains
being penalized in the presence of bulk IO.
Support for network QoS is in pretty good shape: The netback driver
can implement token bucket rate limiting (``x KB every y
microseconds)'''') on any virtual interface to enforce simple
max rate
control. More complex queueing and scheduling strategies can be
implemented simply by invoking Linux''s existing iptables and
traffic\_control facilities.
Some control over disk IO bandwidth is now possible using Linux''s
inbuilt CFQ IO scheduler. Using `ionice'' it is possible to set the
relative priorities for disk accesses. More experimentation is
required to determine whether this will prove to be a sufficient means
of control, or whether some further controls (e.g. implemented in
blkback) will be necessary to provide xen admins with the features the
require. Disk scheduling is a well-known thorny research problem, and
best driven by user requirements.
\section{HVM (fully virtualized) Guests}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
As CPUs with Intel VT and AMD-V support come to market in volume,
``HVM'''' guest support is now a critically important core
feature of Xen.
The current support we have today is ``OK'''', but we know we
can do a lot
better, and really make the new hardware fly. This involves
substantial surgery to a number of key subsystems, but we seem to be
making good progress.
\subsection{Shadow Pagetables}
One of the most important and complex subsystems in Xen is the shadow
pagetable code. For paravirtualized guests, this is typically only
used when guests are undergoing live relocation, but for HVM guests it
is turned on the whole time and is critical to performance. The code
has to support a number of different modes of operation, and deal with
the differences between guest and host page table levels -- the code
supports 2-on-2/3/4, 3-on-3/4, and 4-on-4.
The current shadow pagetable implementation is large and very complex,
partly as a result of having been hacked on by a lot of different
people each addressing their own requirements. The algorithm itself
isn''t too bad, but we''ve now accumulated a lot of profile data
from
different OSes to enable us to do a better job this time round. We
have designed a new algorithm and have embarked on a complete rewrite
of the code, which will take some time complete. Because of the shadow
pagetable code''s importance, testing it is a major task as the test
matrix of different OS versions and configurations is vast. We will
need to run the two code bases in parallel for some period of time,
either in different trees, or possibly in a single tree with a boot
option. We have a list of optimizations and heuristics we intend to
add to the new implementation once the core is stable. In the
meantime, there are various folks continuing to fix and optimize the
code currently in the xen tree, which further helps inform the new
design.
\subsection{QEMU}
QEMU has proved to be very helpful for providing Xen HVM guests with
emulated IO devices. However, Xen''s current
``qemu-dm'''' code has
diverged quite heavily from mainline qemu, which means that we can''t
as easily capitalise on enhancements made to mainline qemu, and also
makes it harder for us to contribute enhancements back to
Fabrice. Fixing this is a priority. We are developing a
re-implementation of qemu-dm that is maintained as a patch queue
against an unmodified snapshot of mainline qemu, and hope to have this
ready for testing soon. Like the shadow pagetable code, this is going
to require extensive testing even prior to inclusion in -unstable.
Catching up with the latest version of qemu has a number of nice side
effects. It gets us Anthony Liguori''s improved VNC server code for the
framebuffer, removing our dependency on the rather obtuse libvncserver
library. Further, it gets us support for emulated USB devices, most
interesting of which is a USB mouse. The USB mouse protocol supports a
mode of operation whereby it provides absolute x,y co-ordinates
rather relative motion events (which the OS usually turns into
absolute co-ordinates using some unknown `black box'' algorithm that
has mouse speed and acceleration parameters). Being able to inject
absolute co-ordinates means that it is easy to arrange for our the
HVM guest cursor to perfectly track the local cursor in the VNC
viewer, regardless of the user''s mouse settings.
Although not immediately on the horizon, there are plans to give qemu
a slightly extended role in Xen. The `V2E'' research work in Cambridge
has shown that it is possible to move the execution state of a guest
from a Xen virtual machine into qemu and back out again. In the
context of the research work the aim was to enable high-performance
taint tracking to be implemented, where the guest ran at full speed as
a xen guest until it accessed a tainted value at which point execution
was moved on to the qemu emulator which was able to monitor execution
closely. At some point later execution would be moved back to Xen. As
pointed out by Leendert, a similar approach can be used for
transitioning into IO emulation, cleaning up the current interface and
possibly providing performance optimizations when many IO operations
are performed in close proximity. This technique also solves a thorny
issue with Intel VT systems, which unlike AMD-V do not provide h/w
assistance for virtualizing the legacy x86 `real mode''. Today, we have
the `vmxassist'' code containing a crude emulator to try and handle
this case, but the code fails in the presence of certain complex 16b
applications, such as MSDOS or the SuSE graphical boot loader. Having
the ability to throw execution onto QEMU which has a complete real
mode emulation would solve this problem.
\subsection{Paravirtualization Enhancements}
Although we can run completely unmodified guest OSes in HVM mode,
there are significant performance advantages that can be achieved by
selectively adding paravirtualization extensions to guests. The nice
thing about this model is that the OS author can boot their OS
unmodified, then incrementally enhance the OS to exploit Xen''s
paravirtualization hypercall API to improve performance, for example,
adding paravirtualized IO drivers, then paravirtualized virtual memory
and CPU operations etc.
The Xen API''s hypercall transfer page assists in allowing the precise
method used to make the hypercall to be abstracted from the guest
(e.g. INT82 in the full-paravirtualized case, VMCALL on VT, VMMCALL on
AMD-V). However, the exact method for installing the page (e.g. by
writing MSRs) and passing in-memory arguments to hypercalls has yet to
be finalized, though there are prototype patches in existence. Making a
decision on which approach to checkin is a priority.
Typically, the paravirtualizing extension that has most impact is
switching from using IO emulation to using PV drivers. This can be
done by taking the core of the existing netfront/blkfront/xenbus
drivers and providing appropriate wrappers to enable them to be built
against a native (non Xen) Linux tree, and thus loaded as modules into
the native kernel running as a HVM guest. PV drivers can similarly be
prepared for other OSes.
\subsection{Save/Restore/Relocation}
One of the key features missing in Xen''s current support for HVM
guests is save/restore. Several subsystems need to be updated to add
this support. We need to exploit qemu''s ability to `pickle''
the IO
state of a guest and be able to pass it down a file descriptor. We
need to add code to Xen to enable the state of all of the
high-performance emulated devices such as the PIC, APIC, IOAPIC etc,
to be read out via a the get\_domaininfo dom0 op. We then need a simple
update to the xc\_save code to work with auto-translate shadow mode
guests. The restore operation requires very similar changes, enabling
device state to be `unpickled'' in both Xen and qemu. Having got
save/restore working, live relocation should actually be relatively
straight forward, just enabling `log-dirty'' mode for auto-translate
mode guests.
\subsection{SMP Guests}
The current HVM code only stably supports uniprocessor guests, though
work is underway to robustify SMP guest support. Xen already contains
APIC and IOAPIC emulation modules, but this code along with the
existing shadow pagetable code lacks synchronization in certain
places. The new shadow mode code should go a long way to solving
this. Work is also ongoing to add ACPI BIOS support, hence allowing
the emulated platform to look like a modern PC. The first aim for
supporting SMP guests is correctness, and once we have that focus on
performance and scalability. Getting fair performance for two and four
way guests should be achievable for many workloads, but this is an
area where paravirtualization really helps. Eventually, it would be
good to support CPU hotplug emulation in Xen.
\subsection{Mid-term goals}
Looking toward the mid-term, there are a number of projects around HVM
guests that would be good to see implemented.
One of the key desires is to move the qemu device emulation out of
domain 0, and run it in a `stub domain'' associated with its HVM
guest. A `stub domain'' is the effectively the execution context
associated with a domain that a paravirtualized guest would normally
use, but is currently unused for HVM guests. Belonging to the same
domain, any CPU time spent executing in the stub domain is naturally
accounted to the guest. Like normal paravirtualized guests, stub
domains are strongly isolated from other domains. However, given the
close relationship with the HVM guest, executing transitions between
the two occur faster than transitions to a secondary domain. Further,
belonging to the same domain means that the stub domain can easily map
memory belonging to the HVM guest. These properties mean that stub
domains are an ideal place to run qemu, providing improved performance,
accurate resource accounting, and surer isolation.
As a user space application, Qemu can''t run in the stub domain
directly, but requires an operating system kernel. The neatest way of
doing this would be to link qemu against `minios'', which is
effectively a library operating system for just this purpose. Since
minios makes use of a broad range of libc calls, it is likely that
minios will take some time to reach the required level of support. In
the meantime, we can just use a xen linux kernel, with a minimal
config to keep the size down. Since protection between user space and
the kernel is irrelevant in the context of running qemu as the sole
application, we could optimize performance by running user-space at
the same privilege level as the kernel, effectively turning system
calls into plain jmp instructions into the kernel followed by a ret to
return.
An interesting thing becomes possible once we have qemu running in
stub domains and interfacing with the HVM guest via the `V2E'' approach
described previously: it becomes quite easy to enable unmodified
guests to be run on CPUs that don''t support VT or AMD-V. Gust
execution in user space would proceed in the normal xen fashion, but
any transition into the kernel would result in the guest being
transferred on to qemu for emulation, which would then transition back
to native execution when the guest exited the kernel. Having the
emulation running in a stub domain is clearly important to allow the
resource spend in emulation to be correctly accounted. Performance
would clearly not be as good with VT/AMD-V approach, but this does
provide a good way for folks with older hardware to experience
unmodified guests on Xen.
>From our point of view, QEMU''s biggest failing is that the devices
it
emulates are quite old, and lack some facilities that could
potentially lead to better performance. It would be nice to have an
emulation of a SCSI HBA, as most OSes typically treat these
differently from an IDE device and make more use of the ability to
have multiple outstanding requests, which is essential for good IO
performance in a virtualized environment. BOCHS has a simple SCSI HBA
emulation, and it may be possible to use this as a starting point.
Emulating a different network device would also offer benefits. A
network device that supported checksum offload, jumbo frames, or TSO
(Transmit Segmentation Offload) could all offer CPU savings in the
guests. Since IO performance is typically dominated by the number of
`vmexit'' operations required, care is required when selecting which
hardware device to emulate to ensure that the corresponding driver
will generate a minimal number of exits. Older hardware is sometimes
better in this respect, since the driver writer knew certain io port
and mmio operations were likely expensive, and thus went out of their
way to avoided doing them. Since such io port and mmio operations
typically generate vmexits in the Xen case, minimizing them is helpful
for performance.
\section{Paravirtualized Guests}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
As stated earlier PV guest support in Xen looks in good shape from a
stability point of view, but we do need to pay some close attention to
benchmarking and tuning, particularly for large SMP guests. There is
undoubtedly plenty of low hanging fruit that can be addressed to
improve Xen performance -- it just hasn''t been a priority given the
state of the competition.
One area that could certainly do with some attention is the 64b Linux
guest ports. The code could do with a full code review, specifically
to remove some of the unnecessary divergence from the i386 xen code,
and to remove some of the modifications relative to native that are
now unnecessary. There are a number of opportunities for investigating
optimizations too. The current code was written assuming that the ``TLB
flush filter'''' found on AMD Opteron processors would become
ubiquitous. Sadly, Intel haven''t adopted this, and AMD have dropped
the feature from future chips. There are a number of places where TLB
flushes occur in the current code where it was expected that the flush
filter would actually annul them. Given that we can no longer rely on
this, some modifications to the virtual memory virtualization
implementation is called for to reduce the number of TLB flushes. One
particular idea that has been mooted is using the global bit in PTEs
to preserve both Xen and User mappings across system calls. This needs
to be implemented and benchmarked. Hopefully Intel/AMD will extend the
x86 architecture with an address-space tagged TLB implementation at
some point, providing a clean solution.
One area that hasn''t received much attention recently is the ``live
relocation'''' feature since the 32b PAE and 64b hypervisor
variants were
introduced. Although the current code seems to work, it''s never been
`productized'' on these hypervisor variants, so shouldn''t be
relied
upon. The new shadow pagetable code being developed primarily for
improved HVM guest support should also help address this concern,
greatly simplifying the current code.
We also need to do some work on the xc\_restore function itself,
modifying it to support lazy allocation of memory to avoid the current
issue whereby when relocating a guest you (temporarily) need to
allocate enough memory for the maximum memory size of the guest even
if its current size is much smaller due to memory ballooning. Further,
we could probably improve guest down-time by a few 10''s of
milliseconds if we did some streamlining of the various hotplug
scripts and python code that do re-plumbing of network and block
devices after a live relocation.
\subsection{PV Block IO}
The current block device protocols and implementations are in pretty
good shape. The work to add a secondary backend implementation called
`backtap'' to support high-performance file-based qcow implementation
has previously been discussed. This blkfront implementation uses the
same guest IO API, and can be used interchangeably with the current
blkback and blktap backends.
One area that does need some discussion is whether the protocol should
be extended to support in-band metadata operations, for example,
enabling the front end to issue something akin to ioctl on the
backend. Such requests could probably be implemented fairly straight
forwardly, as a special variant of a write operation that also returns
data in-place. We would need to have well defined meanings and
enumeration of these ioctl-like operations since there may be entirely
different operating system kernels at either end of the block protocol
device channel. However, for `occasional'' ioctl operations it might be
best just to use the xenbus control channel between front and
backend. This can also be used for passing messages the other way. In
particular, we should use this protocol to handle removable media
change events better than we currently do. This mechanism could also
be used to provide notification of virtual disk size changes, enabling
an on-line resize tool to be invoked to provide seamless growing of
virtual disks.
One extension that has been requested by at least one proprietary
filesystem vendor is the ability to return metadata with both read and
write responses (rather than just noting completion). This meta data
is interpreted by the file system and used to optimize certain
journaling operations. These extensions are currently considered
rather specialized an probably mandate a separate blk protocol.
Another idea that has been mooted is supporting a scsi-level device
channel protocol, and hence scsifront/back drivers. This would be
useful for controlling more `exotic'' devices that need a richer set of
operations than just read/write/barrier, e.g. a tape drive. It does
introduce a whole load of fairly pointless scsi command creation and
parsing at either end for the normal read/write case, which may end up
resulting in measurable overhead. Experimentation is required.
\subsection{PV Network IO}
One key area that needs some performance tuning is the paravirtualized
network device driver. The current drivers use a page-granularity
protection mechanism to provide good containment and thus minimal
trust between the frontend and backend drivers, but this does exercise
the page protection mechanisms pretty hard.
There are a number of fairly obvious extensions to the network device
channel protocol that should yield useful performance improvements.
Some of these could be implemented in a backward compatible fashion by
using spare fields in the current protocol, but at some point we''re
likely to want to define the net device channel v2 protocol. This is
not really a big deal as the xenbus control plane is there to ensure
that we get the right drivers bound at each end of a device
channel.
One cleanup that is required is to modify the way that the checksum
offload protocol is implemented, allowing the offset of the checksum
in the packet to be specified. This should clean up some of the
current fragility we have in domain 0 when checksum offload is enabled
and certain less common protocols are used.
Another useful improvement to the current code would be to add
a flag to the data area of skb''s that have been the subject
of decrypt-in-place operations (e.g. for IPSEC VPN''s), and only
selectively scrub the flagged pages when adding buffers to the free
queue.
The current driver uses page flipping as the only mechanism for
transferring data over the device channel. The device channel
mechanism is actually quite flexible, and with a simple bit of
refactoring, it should be possible to give the hypervisor the option
of either copying the packet or flipping the page, making the decision
based on the operation size. The copy operation may become
particularly attractive if high-performance multi-threaded copy
engines start becoming integrated in to northbridges, as has been
speculated in some quarters.
Another possibility worth investigating is the use of a semi-static
shared memory buffer between the front and backend i.e. the grant
table mappings are set up in advance for the buffer and
left in place rather than being cycled. On the receive path, netback
could directly copy packet payloads into the shared buffer from the
hardware receive buffer, and then netfront could directly copy
out of the shared buffer and into a local skb (2 copies, but no
hypercalls).
It would be nice to eliminate the copy in netfront, but this is quite
hard as the skb could end up queued for an arbitrary amount of time in
an application socket buffer, hence consuming resource in the shared
buffer. We could grow the shared buffer if we run out of space, but
there''s no real bound to the size that would be needed and the number
of in-flight buffers we''d have to track. Possibly some adaptive scheme
that uses copying for buffers that are likely to be long lived or when
we are low on shared buffer resource would be possible. However, the
downside of this approach is that guests have to invest more trust in
their backend domains as they retain writable mappings to the packets
sitting in kernel buffers. A buggy or malicious backend could modify
the contents of a packet buffer after the guest kernel had validated it,
and could very likely cause it to crash.
There are also performance enhancements to be achieved through
supporting some of the higher-level features provided by modern server
adaptors. Jumbo frame support would clearly be useful, both in domain
0 and from guest VMs, but many Ethernet installations still use a 1500
byte MTU, so couldn''t benefit. TCP Segmentation Offload (TSO) for the
transmit path would be useful, whereby the frontend passes the backend
a very large `super packet'' for it to segment and transmit (ideally,
passing it to the NIC to do the segmentation if we can get the super
packet through the bridge and routing code). Even if we end up having
to do the segmentation in the backend we''re still better off than we
were passing multiple packets as we''ve reduced overhead and made it
easier to batch work across the device channel.
To support TSO, we need to extend the device channel descriptor format
to flag packets that should be segmented. Since super packets are
unlikely to be physically contiguous, we also need to be able to
support data fragments in the descriptor format. This should be a
relatively simple extension, adding chaining to netring descriptor
entries.
In future, we should consider how we might export byte stream (TCP)
data between domains in a high performance fashion. This would be
particularly useful between VMs running the same machine, or when TCP
Offload Engines (TOE) in hardware NICs become commonplace. Similarly we
should have an extended interface for exporting RDMA (Remote DMA)
capabilities.
Examining oprofile traces of Xen systems under heavy network load it
looks like the network bridge code running in domain 0 takes a
surprisingly large slice of the CPU. It''s quite possible that
we''d do
better with some cut-down streamlined bridge code that just handles
the common case that typical Xen installations use it under. At the
very least, we need to do some investigation into why the current
bridge code shows up in profile results as high up the ranking as it
currently does.
The current netfront/netback approach is geared toward having domain 0
(or another domain) acting as the switch/router for packets between
other domains. There are some circumstances where it may be more
efficient to create virtual `point-to-point'' links between VMs,
enabling them to communicate directly without going via a software
switch/router. This would be akin to connecting two netfront devices
together, allowing very high performance networking between two VMs on
the same machine. This could be used in applications where there is
effectively a processing pipeline of data being passed between
VMs. Were one of the VMs in the pipeline to be migrated to a different
machine, the netfront on each end of the point-to-point link could
revert back to being connected to a netback driver, fulfilling the
connection via the external network, albeit at lower bandwidth.
While adding all these extensions to the protocol we should remember
that these same paravirtualized drivers are used to provide
paravirtualized IO within HVM guests, and ensure that they work in
both scenarios.
\subsection{Client Devices}
Although most Xen deployments are currently targeted at server
machines, improving support for desktop and laptop usage is clearly
important, not least to encourage developers to run xen on their main
machines. Having excellent support for client devices such as USB and
the frame buffer is clearly required, along with good power management.
Way back in the days of Xen 2 we had support for passing control of
individual USB devices over to a guest VM. However, although
apparently stable, the implementation wasn''t ideal and was never
updated to Linux 2.6. The best you can do today on Xen 3 is to assign
a whole USB host controller to a guest using PCI pass-through, rather
than individual hub ports as before.
There have been patches floated for adding this support back to Xen 3,
but there''s never been quite the impetus to get consensus and get
something checked in. The situation has recently become more
complicated with the USB-over-IP code now appearing in Andrew
Morton''s Linux tree, perhaps suggesting that the design should be
revisited.The TCP transport would be replaced with a reliable
byte-stream device channel transport to avoid the need for network
configuration. It would be good to return USB support to Xen 3 as soon
as possible, as this will solve a number of requirements around audio
virtualization, mouse, scanners etc.
For PV guests, Xen currently only supports a virtual serial
console. If a graphical console is required the guest needs to run its
own networked frame buffer console, typically Xvnc. This is less than
ideal as it requires networking to be working in the guest before the
console can be used. Serial console serves perfectly well for boot,
though some users get confused by the differences between a serial
console and a typical Linux Virtual Terminal (VT).
The best solution to this is to implement an in-kernel fbdev
paravirtual frame buffer driver, which uses a shared memory device
channel to make the frame buffer available to domain 0, where it can
either be rendered locally, or converted to a network frame buffer
protocol. This is precisely what happens with the console of HVM
guests today, so will also help unify the `look and feel'' between PV
and HVM guests. Having the PV framebuffer as a kernel fbdev means that
it will be able to emulate a text mode and display messages from
fairly early in the boot sequence, again making operation closer to
native.
In a basic implementation, the framebuffer control software in domain
0 would run periodically and either copy the shared memory region to the
local display window, or generate the appropriate VNC updates. Scanning
the whole frame buffer is obviously inefficient, particularly as there
are frequently no updates, or the updates are quite localised (e.g. a
flashing cursor). The current HVM frame buffer code implements a
scheme whereby page-level write-protection of the framebuffer is used
to trap updates, and hence provide a (very) rough indication of which
areas of the frame buffer need scanning for updates. Since this code
primarily uses VNC as a backend, the current framebuffer contents is
compared against a snapshot to enable a more compact network encoding.
The PV framebuffer should ideally share much of this code. Rather than
using a page-fault based approach, if the guest maps the framebuffer
once from a single set of pagetable pages, it may be possible to get
better performance using `dirty'' bits on PTEs rather than taking write
faults.
Achieving decent 2D graphics performance really requires more help
than can be achieved simply by monitoring page-level granularity
updates. Ideally, the framebuffer backend code would be provided with
accurate bounding box update rectangles, and explicit `copy region''
and `fill region'' commands (along with a separate hardware cursor
rather than just rendering it into the framebuffer). In the HVM guest
case, this could be achieved by emulating a graphics card which
supported (and whose common OS drivers supported) copy and fill
operations. In the PV guest case, the kernel fbdev interface is not
rich enough to supply this data, so this implies that we will need to
write an Xserver driver module that supplies this data to the backend
via a side-band shared memory interface. Requiring modification of the
Xserver to achieve decent graphics performance is unfortunate, but
there is precedent -- this is the approach VMware have taken.
Anthony Liguori and Markus Armbruster are working on a PV framebuffer
patch that addresses many of these issues and we hope to commit it
shortly.
Achieving decent 3D graphics virtualization requires a substantially
higher-level interface than that for 2D. With both the Xserver and
Microsoft window systems moving in the direction of using 3D rendering
even for desktop graphics, 3D graphics is becoming mainstream and not
just the preserve of games and CAD/CAM. We will need to investigate
drivers that encapsulate and transport OpenGL and/or Direct3D commands
into backend domains where they can be rendered by the 3D graphics
hardware. There are already a couple of projects that are
investigating this is the context of OpenGL: Jacob Gorm Hansen''s work
presented at the last Xen summit, and a new project at CMU/Toronto
based on Chromium.
\subsection{Smart IO devices}
There is no denying that IO virtualization in software incurs extra
latency and CPU overhead relative to native. For applications that
require the best possible performance, some help from the IO hardware
is required. For over a decade their have been specialist `smart''
network interfaces available that have had the necessary hardware
support to be accessed directly from user-level applications in a safe
and protected fashion. Most of these interfaces have been targeted at
the High Performance Computing market, designed for doing low-latency
message passing, but some newer interfaces also support standard
TCP/IP/Ethernet protocols. The requirements made of a network device
to be able to provide safe direct access from guest virtual machines
is very similar to those for providing direct access to user-level
applications in a traditional non-virtualized environment. We
anticipate many of these smart network interfaces becoming quite
mainstream over the next couple of years as virtualization becomes
ubiquitous. The same principles can also be applied to storage access,
but in our experience the benefits are less pronounced.
Work by IBM has already demonstrated direct VM access to Infiniband
hardware on xen, and the code for this is available in the
ext/xen-smartio.hg tree on xenbits. In future, we expect to see
support added for the Level5 and NetXen (previously known as UNM)
smart NICs which are more like traditional Ethernet interfaces.
Typically, the main device driver is run in domain 0, which is
responsible for allocating the NICs resources to other VMs and hence
allowing them to map page-aligned control regions of the PCI device''s
address space into their own virtual address space. The main driver
also controls what memory pages each of the virtual NICs is allowed to
issue DMA operations to/from. Guest VMs use a small unprivileged
driver to operate the virtual NIC''s free/receive/transmit queues via
their control area. The NIC typically also supports some QoS traffic
shaping of outbound traffic from a virtual NIC, and may support more
sophisticated receive demultiplex and inbound/outbound firewalling
options beyond simple layer-2 demultiplexing. Having such features
implemented in hardware is important as users typically don''t want to
give this up when switching from the current s/w solution.
Before the code can be included in the main xen tree we need to have
decided the interface through which these smart NICs interact with
Xen''s memory management to ensure that a guest requesting DMAs to a
page is actually the owner of that page. Since most OSes tend to
re-cycle network buffers it is usually the case that network packets
are received into and sent from a relatively small and static pool of
memory. This is quite a simple scenario that can be dealt with by
pre-pinning the buffers with the NIC and with Xen. Storage poses more
of a challenge as the pages used for DMA will be spread throughout the
system, particularly for writes (though writes are less latency
sensitive). Clearly, making decisions on this interface should be
strongly influenced by the support planned for chipset IOMMUs.
Another area where care needs to be taken is consideration of the
interaction between VM relocation and smart NICs. In general, it is
not safe to save/restore/relocate a VM that has direct access to
hardware. Given that access to the interface is under the control of
the domain 0 main driver, it is typically possible to arrange for this
operation to be safe. However, relocating a VM to another physical
machine would require that the destination machine had the same smart
NIC hardware, which is inconvenient. We are considering the
possibility of having a pluggable driver architecture that could enable
the domain builder to `slide in'' a different low-level driver suitable
for the new machine.
\subsection{PV Filesystem-level Virtualization}
Block-level IO virtualization provides an abstraction for accessing
storage that users will be very familiar with. However, a
filesystem-level abstraction offers a number of interesting
possibilities too. This would work rather like a network file system
such as NFS or OpenAFS, but without the need for networking as a
shared memory device channel transport between fsfront/fsback would
employed. Thus, ``xenfs'''' would enable one domain to export a
subtree of
its file system to be imported and mounted by another virtual
machine. A range of different semantics could be implemented,
including a typical coherent shared file-space, or copy on write.
The xenfs approach is quite interesting as it provides a very high
performance mechanism for guests to share data in a coherent fashion
via shared buffer cache mappings to the same file. A shared buffer
cache could also avoid some IO, and yield memory savings. Implementing
this for PV guests requires use of a new type of page fault,
``copy-to-write'''' which is different from
``copy-on-write'''' in that the
all mappings to the immutable old page must be updated to the new
copy. Fortunately, an implementation of CTW faults for Linux has already
been developed by the embedded systems community, who need it when
mapping pages stored in Flash memory.
Mark Williamson is working on the XenFS implementation.
\section{Core hypervisor}
%%%%%%%%%%%%%%%%%%%%%%%%%
One of the key features we''d like to add to Xen soon is the ability to
run a mix of 32b and 64b paravirtualized guests on a 64b hypervisor
(just like you can with HVM guests already today).
Because of the similarity in pagetable formats, it should be possible
to run 32b PAE guests on a 64b hypervisor pretty straightforwardly and
with good performance. We just need to provide a `compat32'' version of
the hypercall table and export a 32b version of the `m2p'' table. Since
the hypervisor can live outside the 32b address space of the guest we
can exploit the code in the Linux port to allow a variable sized
hypervisor hole to enable us set the hole size to zero, giving the
guest the full address space. Supporting non-PAE guests is also
possible, but would require the use of shadow pagetables to convert
between the guest and host pagetable formats. Given the resulting
performance hit, it''s probably best to just stick to PAE guest
kernels.
Work on adding NUMA support to Xen is important now that integrated
memory controllers on CPUs are commonplace. The goals for this work
have been discussed earlier in this document.
Right now, Xen has no support for allowing guest kernels to exploit
superpage mappings to access 2/4MB machine-contiguous memory
regions -- we always factor such mappings into multiple potentially
sparse 4KB mappings. This has the potential to impact TLB usage for
some workloads [NB: it is worth noting that both shadow-mode and
direct-mode pagetable implementations mean that it is typically
necessary to protect kernel mappings on 4KB page granularity so it is
typically not possible to allow kernel use of such mappings.]
Xen''s memory allocator is already capable of managing and allocating
contiguous memory chunks. However, we would need to add some guest
accounting to control the number of multi-page contiguous regions a
guest is allowed to claim, otherwise we have the potential for one
guest to hog all the contiguous chunks to the detriment of
others. Such accounting is useful today even without superpage
support as privileged guests can use machine-contiguous regions for IO
purposes, and it would be good to bound the number of these.
Adding superpage support for mappings of user-space memory for HVM
guests is pretty straightforward, though it will always be necessary
to be able to factor one of those mappings later if maintaining the
invariants of the shadow pagetable algorithm requires it. Support for
direct-mode paravirtualized guests should also be possible, though is
made slightly more complicated by an annoying quirk of the x86
pagetable format that means that linear mappings interact poorly with
superpage entries as it is not possible to generate a trap if a super
page PTE is accessed as though it is an L1 entry. Places in Xen where
we access the guest pagetable via a linear mapping will need to be
audited. On a 64b hypervisor there should be fewer of these as using
the direct 1:1 mapping is typically preferable.
Power management is also a concern for the mid-term. We can certainly
put idle processes into a deeper sleep state than we currently
do. Looking even further out, we could also even adapt the Xen
scheduler to make CPU clock frequency scaling decisions, and
distribute load across processors to minimize power requirements.
Getting whole-system (as opposed to guest) suspend and hibernate
support would be a useful feature for laptop users. This shouldn''t
actually be too hard, to the extent that Linux supports the function
running native on a given system. We will need to add stubs in Xen to
do the final stage power down and following resuscitation.
IOMMUs look set to become common on future x86 server platforms, which
offers a number of benefits for Xen. IOMMUs are clearly useful to
ensure protection when assigning an IO device to a VM: without one a
malicious or buggy guest could destablize the system or read data
belonging to other VMs by instructing the device to DMA to/from memory
pages other than those it owns. As well as providing protection, most
IOMMUs also provide translation of DMA addresses. This means that they
can be used to enable HVM guests to be delegated direct access to a
hardware device as well as PV guests. Note that save/resume/relocate
operations are likely not possible on guests with direct hardware
access, unless the hardware and driver has been designed for this
purpose.
Most IOMMUs typically don''t support the fault-and-fixup style of
operation that is common with CPU memory management: Any lookup
failure is difficult to recover from, and likely involves resetting IO
buses and devices, and likely results in lost IO operations. Xen must
be able to recover from such situations.
In the case of a HVM guest that is not actively co-operating with Xen,
Xen has to maintain an IOMMU data structure containing the full set of
guest physical pages the guest may wish to instruct the IO device to
access. It must ensure that each of those pages is pinned and hence
it''s ownership (or mmu type) can''t change. Before a page can
be
unpinned and released (for example, by the balloon driver), xen must
remove it from the IOMMU pagetable structure, issue a flush or
invalidate, and wait for confirmation the flush has completed. On a
processor supporting nested pagetables, it may be possible to share
the guest-physical to machine frame translation pagetable with the
IOMMU.
For PV guests (or HVM guests with PV extensions), the guest can be
more actively involved with co-ordinating what pages are accessible
via the IOMMU. The simplest mechanism for doing this would be to use
the hooks provided by the kernel''s DMA mapping/unmapping API and have
these operations call down into Xen to update the IOMMU
appropriately. On x86 this strategy is likely to be quite costly as
there is little batching to amortize the hypercall to do the mapping,
though unmapping can potentially be done lazily provided the pages
remain pinned.
Driver domains need to perform IO on behalf of other domains, and the
grant table mechanism gives them the ability to create temporary
mappings to read/write data. To be able to deliver data in a zero copy
fashion, page grants need to be extended to the IOMMU. The grant table
mechanism was designed with this in mind, so it''s a relatively
straightforward extension. When calling into xen to map a grant
handle, a guest can specify whether the page should also be added to
the IOMMU. The grant unmap operation can operate in a similar fashion.
The only potential issue with this approach is that at the point that
the map operation is performed the device that is going to be doing
the IO may not yet be known (for example for network transmit we need
to do the map operation before we can inspect the packet header and
determine the destination interface). In most situations, it probably
makes sense for all devices under control by the same driver domain to
share the same pagetable structure, so this is not an issue. If this
is not the case, mapping can be deferred to the kernel''s DMA mapping
functions, but there is likely to be less batching to amortize the
hypercall cost.
In 32b x86 Xen, address space is at a premium. This leads to the
notion of xen heap pages and separate `domain pages''. The former may
be accessed by Xen rapidly in any context, whereas domain pages must
be mapped dynamically on demand. On x86\_64 this distinction is
redundant as Xen has a 1:1 mapping of all physical memory, and hence
domain pages may be accessed efficiently. However, the current x86\_64
code inherits the statically sized heap from 32b xen. As an immediate
fix, this heap is arguably too small for a large x86\_64 machine, and
limits the number of VMs that can be started (the limit is lower than
on 32b as the domain structure for VMs is larger on 64b builds). A
better solution might be to unify the xen and domain heaps on 64b
builds.
IBM have led the excellent work to add fine-grained access control
mechanisms to Xen''s low-level interfaces. However, the current dom0
control interface has a very simple `flat'' notion of privilege, and
extending this to allow more flexible delegation of control over
guests would certainly be desirable for some deployment scenarios.
Today, there already exists a notion of a bitmap of privilege
capabilities that a guest has. (Note that this is orthogonal to the
sets of physical resources it has control over such as ranges of
machine address space, io ports etc). The current capability set is
rather small, and could no doubt be made more fine grained.
More interestingly, it would be useful to be able to delegate
privilege such as to be able to grant a domain permission to perform a
certain privileged operation on some specified other domain or group
of domains. This leads naturally to a hierarchical model of
domain resource allocation and permission, for example allowing a
domain with only a very restricted privilege capability to create a
new domain by carving it out of its own resource allocation. It would
then have full control over this domain, allowing it to destroy it,
pause it, map its pages, attach a debugger etc.
>From Xen''s low-level `datapath'' point of view we want to
flatten this
hierarchy to keep the privilege check operations as simple as
possible, with only the control operations having to worry about the
extra complexity. Citing the example in the previous paragraph of
having one domain build another, this should be quite achievable as
some care is already taken to have the domain builder use standard
unprivileged interfaces.
\section{Testing and Debugging}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Users require Xen to be a rock-solid stable system component,
achieving greater stability than the OSes which run on it. Generally,
we don''t do too badly on this front. We''re fortunate to have a
relatively small and tight code base, coupled with significant
investment in testing by a number of parties.
The tip of the -unstable tree is exercised daily by IBM, Intel and
VirtualIron as well as XenSource. The XenSource test reports are
viewable on the web at http://xenbits.xensource.com/xenrt, while other
reports go to the xen-devel list. The continuous testing provides a
good way of monitoring the progress toward stability of new features,
and for flagging regressions.
In fact, most potential regressions never make it out into the
-unstable tree, as they are picked up by automated testing in the
staging tree, which gives changesets a grilling on three different
machines (32b, PAE, and x86\_64) before pushing the changesets out to
-unstable. Hard-core developers wanting to see checkins as soon as
they happen can do so by subscribing to xen-staging@lists.xensource.com.
Prior to the 3.0 release we released a special testing CD and
encouraged users to download and run it on their hardware then upload
the results. The CD provided booted a native Linux kernel followed by
32b, PAE and x86\_64 kernels (as appropriate for the hardware), running
a whole series of tests and benchmarks under each and recording the
results to a log file which was then uploaded. The test CD proved very
useful identifying machines Xen struggled on, and helped us get many
of the issues fixed prior to release. Post release, the 3.0 demo CD
has proved useful in debugging various platform issues reported by
users, enabling them to collect and supply developers with data from
both native and xen kernel boots.
One avenue we are investigating to help find bugs is submitting Xen to
the OSS code scanning programme run by Coverity with funding from
DARPA. Coverity''s tools have proved useful in finding bugs in a number
of other kernel-level projects, so it makes sense to investigate. The
tool will likely generate a long list of possible issues, which will
require a concerted effort from the community to investigate and
classify, and fix the subset that are real bugs.
Even with the best possible test and QA programme, some number of
in-field crashes will be sadly inevitable. In these circumstances,
it''s important that developers can gather as much information as
possible from the incident. For user-space incidents, `xen-bugtool'' is
useful for collecting log files and system details. For more serious
failures of domain 0 or xen itself we can''t rely on things getting
logged in standard files, and need support for taking a system core
dump. Horms is doing great work to get kexec working in domain 0,
meaning that we will be able to use kdump to write out a system core.
Dumping a full copy of system memory can be quite time consuming on a
machine with lots of RAM, so we will possibly have to consider putting
a little more intelligence in the dump routines to harvest the most
useful information e.g. a quick dump that concentrates on CPU register
state and stack information for xen and dom0, and a more comprehensive
dump that collects all xen and dom0 state and just register and stack
info for other guests. We certainly need tools to help pick apart
these dumps and turn them into a form that gdb can load to assist
examination.
For xen developers there are a number of debugging aids that can
called on. There is a serial gdb stub that can be connected to
remotely and used to examine both xen and domain 0. Since this halts
the system while the debugger is connected, it typically can''t be used
on production systems. One frequently useful tool is the debug
console, which by default is accessed by hitting ctrl-A three times on
the serial console. Hitting `h'' gives a help menu of features the
console can perform. It''s main use is for diagnosing the state of the
system in event of a hang. Register dumps can be used to see what all
the CPUs are doing, whether guests are servicing interrupts etc. The
console also provides access to the software performance counters and
other
statistics.
Debugging guest VMs can be achieved using Xen''s gdbserver support.
When gdbserver is started in dom0 against a particular VM, it
effectively implements gdb serial stub functionality on behalf of the
guest, enabling gdb to connect to the gdbserver via a TCP
port. gdbserver has recently been extended to support both PV and HVM
guests. If getting register and stack state for all the VCPUs from a
guest is all that''s required, the `xenctx'' command is quicker
than
firing up gdb.
When guest VMs crash we currently have the option to preserve the VM,
enabling debugging via gdb or xenctx. However, it would be useful to
have the option to write out a core image of the guest VM. This is not
quite the same as doing a VM save-to-disk operation, as we would like
to write the image out in a form that gdb could then inspect. The old
xen 1.2 tools had basic guest core dump support, and it would be good
to see this feature back in Xen 3.
\section{Misc}
%%%%%%%%%%%%%%
We need to add a hypercall to xen to allow a guest to extend the size
of it''s grant table. This should be straightforward, but in its
current absence is the restriction which leads to the current ``max 3
VIF''s per guest'''' limit. Actually, this restriction
could also be
solved by making the allocation of grant table entries to a VIF
dynamic, as already happens with the blkfront driver. We should
implement both strategies forthwith.
It would be useful to add PV extensions to guests to assist in taking
consistent filesystem snapshots, for example, when creating ``template
VMs''''. The obvious way of doing this would be to extend the
xenbus
mechanism used to deliver shutdown requests and `magic sysrq''
operations. Adding the ability to issue ``sync disks and
pause'''' and
``remount filesystems read-only and pause'''' would be useful.
We need to make a few simple extensions to the control API to allow
the set of CPU feature flags exposed to a guest VM (both HVM and PV)
to be ``cooked'''' such that features may be hidden. This is
useful in a
heterogenous CPU environment where it may be desirable to only allow
guests to see a lowest common denominator set of flags such that
save/restore/relocate images remain portable.
The linux guest code currently implements a strict priority ordering
over how it services pending event channels. We may wish to replace
this with a scheme that uses round-robin servicing of events within
each group of 32 event channels, thus providing a hybrid approach that
supports both prioritization and fairness. This is purely a guest
issue, and the current strict priority scheme doesn''t seem to cause
any problems today.
Currently, each PV guest is allowed just a single virtual serial
console device. Although once we have PV framebuffer support (and
associated virtual terminals) virtual serial console support will be
less important, it would still be nice to support multiple consoles
per guest. This should be relatively straight forward, requiring
updates to the console frontend driver, xenconsoled, and the tools.
When using `xm mem-set'' commands to control the amount of memory in a
guest its currently quite easy to set the target too low and create a
`memory crunch'' that causes a linux guest kernel to run the infamous
`oomkiller'' and hence render the system unstable. It would be far
better if the interaction between the balloon driver and linux''s
memory manager was more forgiving, hence causing the balloon driver to
`back off'', or ask for more memory back from xen to alleviate the
pressure (up to the current `mem-max'' limit). The hard part here is
deciding what in the memory management system to trigger off -- at the
point where the oom killer runs the system is typically already
unusable, so we want to be able to get in there earlier. Seeking
advice from Linux mm gurus would be helpful.
When PV guests boot, the kernel and initial ram disk images must be
supplied to the domain builder. This is not unlike what happens when a
physical machine boots using PXE. However, from a maintenance point of
view it is very convenient to store the kernel and initrd in the guest
virtual disk image, where it can be kept in-sync with its kernel
modules. This has led to a couple of different schemes to read the
kernel and initrd from out of the guest filesystem as part of the PV
guest domain building process. pygrub uses libext2/libreiser to read
the images out of the guest filesystem, whereas domUloader mounts the
guest filesystem in dom0 to extract the files. The latter suffers from
potential security issues in the presence of maliciously crafted
filesystem images, but is otherwise simpler to set up.
A better solution to this problem would be to use a bootloader run
within the guest domain. This could be done using a cut-down linux
instance that kexec''s the final guest kernel, or by porting a
bootloader such as grub to be able to use the PV net/block
devices. Grub2 has quite wide filesystem support and is considerably
easier to work on than the old grub code, so adding these drivers may
not be that difficult. Grub supports a wide range of filesystems,
including UFS that used by Solaris. [NB: is UFS supported in Grub2?]
The save/restore/relocate in Xen provides almost all we need to be
able to take copy-on-write snapshots of VM''s memory, to be used for
rollback or to checkpoint long running jobs (if the guest is
communicating with other machines then the wider effect of such a
rollback must be considered). To be able to support checkpoints, we
need to extend the tools to coordinate snapshotting of virtual disks
with taking the execution state snapshot. As well as enabling such
checkpoints to be initiated from the control tools, perhaps providing
the ability to trigger them from within the guest would be useful too.
A natural progression from supporting checkpointing would be to
enable ``VM forking''''. Rather than creating a read-only
checkpoint,
the VM effectively becomes cloned, running in a different domain,
writing to a CoW snapshot of the disk. For VMs that have network
access a mechanism is needed for enabling the IP configuration of the
guest to be updated for the cloned VM. It should be possible to add
code in the PV `resume'' path to change the IP address and update
listening sockets while closing open connections.
Xen currently lacks support for devices that require ISA DMA (DMA
below 16MB). Although ISA devices are rare these days, some PCMCIA
cards have the same restriction (particularly older WLAN cards),
creating problems for some laptops. We need to assess whether adding
such support is going to be worth the effort or not.
Adding support for call graph support into xen oprofile would be
useful. This would involve capturing the first few items on the stack
as well the EIP (though there would be issues when frame pointers are
missing).
Hardware performance counters are currently considered to be
system-wide, which works great for xen oprofile, but prevents guests
from profiling themselves. Adding support to allow performance
counters to be efficiently contexted switched between domains would be
useful (though use in this manner would be mutually exclusive with
system-wide use).
Although efforts are made to ensure the security of the xen hypercall
API, it wouldn''t hurt to do a complete code audit: it is critical that
unprivilged guests should not be able to access data that doesn''t
belong to them, or crash the system. Efforts should be made to bound
the scope of ''denial of quality of service'' attacks too.
A useful tool to help test the integrity of the xen guest API would be
a xen equivalent of the ''crashme'' tool used for testing the
Linux
process API. The tool would attempt to upset Xen by operating like a
normal guest but using random numbers to perturb hypercall arguments,
make ''difficult'' corruptions to pagetables and other
structures etc.
(e.g. mutually recusrsive pagetables would be ain interesting one).
Finally, its about time we did another iteration on the user manual,
wiki and other documentation to ensure its up to date and relevant.
\section{The IA64 ports}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%[insert info supplied by maintainers]
The IA64 port is making excellent progress with contributions from HP,
Intel, Fujitsu, Bull and VALinux to name but a few. Base performance
is currently excellent, incurring an typical overhead of around 2\% on
native. Multiple Linux domains are supported (including support for
virtual SMP), and clean shutdown has been completed. In addition the
source code has been reorganized to produce a clean abstraction layer
at the source level; with this patch applied, the resulting kernel
binary can be run both under Xen and on bare metal.
Xen/ia64 is also getting close to feature parity with the x86
ports. Virtual block device using the standard backend/frontend
model has been integrated, and the same set of control tools
and commands are used to manage the system.
There is also support for VT-i --- the hardware virtualization
e-technology for IPF --- which allows the running of completely
unmodified guest operating systems along-side paravirtualized
domains. Once again, the control tool set has been aligned with
the VT-x model to maximize compatibility.
In terms of hardware compatibility, it has been extensively
tested to operate on the following configurations:
\begin{itemize}
\item Intel Tiger4
\item HP rx2600/rx1600/rx2620 (and likely any HP zx1 based system)
\item Bull NovaScale 4000 and 6000 systems, with 5000 to follow
\item Fujitsu Primequest series
\end{itemize}
In terms of future work, the main areas are:
\begin{enumerate}
\item Completing/stabilizing the work to give each guest a
virtual physical address space;
\item Implementing save/restore and migration;
\item Providing support for driver domains; and
\item Additional stability / performance testing and improvements.
\end{enumerate}
\section{The PowerPC Port}
The PowerPC work, led by IBM, is focusing on the PowerPC 970 processor
which includes hardware extensions designed to support paravirtualized
operating systems. An additional parallel development task is looking
at getting Xen running on 970s with hypervisor mode disabled (e.g.\
Apple G5 systems).
The port doesn''t use the standard split driver model yet, but
is running guests. We hope that the code can get
merged into -unstable in the next couple of months; various changes to
support this forthcoming merge have already been incorporated (e.g.
the use of {\tt copy\_from\_guest()} which on PowerPC uses
physically addressed scatter/gather lists). Further changes to common
code will also be required, which may modify the dom0 API, and so
we need to ensure that people have advance warning before any such
changes go in.
Power Xen currently only runs on the Maple 970 processor evaluation
board, but support for IBM JS21 blades and other 970-based machines
will follow soon.
\section{Aims for next two releases}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{3.0.3 aims; end July to synchronize with FC6 freeze}
\begin{itemize}
\item new CPU scheduler
\item cow file-backed virtual hard disk support
\item qemu-dm updated to latest qemu version, stored as patch queue
\item basic NUMA memory allocator support
\item kexec support
\item xend VM life-cycle management
\item new shadow pagetable code?
\end{itemize}
\subsection{3.0.4 aims; end Sept}
\begin{itemize}
\item new xml-rpc control API
\item simple storage management in xend
\item qemu `v2e'' integration
\item PV network virtualization improvements
\item PPC merge?
\item PV USB support ?
\end{itemize}
\section{Task priority list}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
{\em Schedule:} \\
1 - 3.0.3: end July \\
2 - 3.0.4: end Q3 2006 \\
3 - Q4 2006/Q1 2007 \\
4 - beyond \\
These priority ratings indicate where various features could be
targeted to land on the roadmap. It reflects where the xen core team
is planning on concentrating its effort, coupled with input from work
that we already know is happening in the community. Obviously other
folk will have different ideas of priority and will scratch their own
itch and submit patches. We should obviously try to iterate in order
to get such patches included ASAP, regardless of the below suggested
schedule.
\small
\begin{tabular}{|l|l|p{10cm}|}
\hline
Sched & Area & Description
\\
\hline
1 & tools & xend VM life-cycle management
\\
1/2 & tools & XML config file and conversion tools
\\
1/2 & tools & standardized xen control API: xml-rpc over
https/unixdomain sockets \\
1/2 & tools & C++/perl/python bindings for control API
\\
2 & tools & simple storage management in xend
\\
3 & tools & revive guest coredump support
\\
3 & tools & split VM relocation operation into two parts
and authenticate \\
3 & tools & DMTF CIM providers
\\
3/4 & tools & Web GUI for Xen
\\
\hline
1 & storage & blktap (or other) support for file-based
virtual disk storage. \\
2 & storage & blktap plugins for common formats
\\
2 & storage & optimized qcow implementation
\\
3 & storage & consider adding write accounting/throttling on
current loop driver \\
3 & storage & support for block IO QoS. Use CFQ and ionice,
or implement in blkback? \\
3/4 & storage & ''ioctl'' support between blkfront/back
\\
3/4 & storage & media change, size change event propagation to
guest userspace \\
4 & storage & consider SCSI level storage virtualization
option \\
\hline
1/2 & network & TCP Segmentation Offload support in device
channel \\
2 & network & checksum offload cleanup
\\
2 & network & hypervisor chooses to copy vs. page flip
\\
2 & network & dynamic allocation of grant table entries;
grant table resize
\\
2 & network & investigate whether bridge code needs to be
''streamlined'' \\
2 & network & jumbo frames support in dom0 and device
channel \\
3/4 & network & investigate static shared buffer approach
\\
3/4 & network & TCP Offload Engine support in device channel
\\
3/4 & network & investigate high-performance point-to-point
link support \\
4 & network & RDMA support in device channel
\\
\hline
1 & xen & extensive benchmarking and perf tuning
\\
1 & xen & CPU scheduler that balances VCPUs, implements
weight \& caps \\
1/2 & xen & initial NUMA mechanism checkin
\\
2 & xen & live relocation tuning, robustification, tools
safety interlock \\
2/3 & xen & support for running 32b PAE guests on a 64b
hypervisor \\
2/3 & xen & improved NUMA policy code
\\
2/3 & xen & add order$>$0 guest memory allocation
accounting \\
2/3 & xen & extend x86\_64 heap size; merge xen and domain
pools \\
3 & xen & investigate bad pre-emption
avoidance/mitigation strategies \\
3 & xen & add superpage support for PV guests
\\
3 & xen & IOMMU support: isolation of devices to
domains; grant table integration \\
3/4 & xen & lazy memory allocation for live relocation of
ballooned guests \\
4 & xen & fine-grained delegation for dom0ops;
hierarchical resource model \\
4 & xen & power management enhancements: CPU sleep, freq
scaling \\
4 & xen & power management enhancements:
suspend/hibernate \\
4 & xen & accounting and billing time IO domains spend
on behalf of guests \\
\hline
\end{tabular}
\begin{tabular}{|l|l|p{10cm}|}
\hline
Sched & Area & Description
\\
\hline
1 & hvm & fix current shadow pagetable code, add
PAE-on-PAE mode, SMP safety \\
1 & hvm & upgrade QEMU version, maintain as a patch
queue \\
1/2 & hvm & rewrite shadow pagetable code to optimize,
simplify \\
1/2 & hvm & finalize interface for making hypercalls from
VT guests \\
2 & hvm & HVM save/restore support; qemu, xen, and tools
changes \\
2 & hvm & basic SMP HVM guest support; ACPI tables,
locking safety \\
2/3 & hvm & change QEMU-xen interface to use the
''v2e''
approach \\
2/3 & hvm & SMP HVM guest performance and scalability
\\
2/3 & hvm & support real superpage mappings for HVM guests
(after adding accounting) \\
2/3 & hvm & implement high-performance SCSI HBA emulation
\\
2/3 & hvm & implement high-performance Ethernet emulation
\\
3 & hvm & live relocation. Add log-dirty support
\\
3 & hvm & move QEMU into a ''stub domain'' linked
against
a linux kernel \\
3/4 & hvm & move QEMU into a ''stub domain'' linked
against
a minios \\
4 & hvm & HVM hotplug CPU emulation
\\
\hline
1 & linux & extensive benchmarking and perf tuning
\\
1/2 & linux & SMP scalability improvements
\\
1/2 & linux & work to get xen in to kernel.org linux
\\
2 & linux & code review of x86\_64 port
\\
2 & linux & investigate proposed x86\_64 optimizations
\\
2 & linux & improve interaction between balloon driver and
page allocator to avoid memory crunch\\
2/3 & linux & support for multiple virtual serial consoles
\\
3 & linux & consider hybrid round-robin/priority scheme to
service event channels \\
\hline
1/2 & client & basic kernel fbdev paravirtual framebuffer
implementation \\
2/3 & client & USB virtualization; investigate USB-over-IP
code \\
2/3 & client & Xserver support for ''h/w cursor'', copy
rect
and fill rect \\
3/4 & client & OpenGL/Direct3D virtualization
\\
\hline
2/3 & misc & support for dom0 kexec/kdump to get a machine
core \\
3 & misc & infiniband direct guest IO support (finalize
interface, merge xen-smartio.hg) \\
3 & misc & support for hiding CPU feature flags from
guests (PV and HVM) \\
3/4 & misc & smart NIC direct guest IO support
\\
3/4 & misc & submit xen for scanning by Coverity tool;
investigate warnings flagged \\
3/4 & misc & tools support for doing auto CPU/memory
resource allocation across VMs \\
3/4 & misc & support to checkpoint/rollback guests
\\
4 & misc & port Grub2 bootloader to net/blockfront
devices \\
4 & misc & investigate ''pluggable driver
architecture''
\\
4 & misc & xenfs filesystem-level virtualization; shared
buffer cache \\
4 & misc & do we need support for ISA/PCMCIA DMA (below
16MB)? \\
\hline
\end{tabular}
\end{document}
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Mark Williamson
2006-Jul-13 13:07 UTC
Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
Just floating this for general comment - For the bootloader work, we should consider Kboot: http://kboot.sourceforge.net/ Kboot is designed to use a "conventional" bootloader to start a Linux kernel running "full" bootloader functionality. This is responsible for loading and kexec-ing the kernel the user actually wants to run. The advantage of this is that all the high level functionality of grub (interactive menus, filesystem support, networking) are now provided by a full Linux environment rather than a large special purpose codebase. Some folks at IBM (Michael included) have been working on an update to this called kboot_sl (system loader): http://merbach.net/kboot_sl/ The zSeries guys want this because their z/IPL bootloader doesn''t have any interactivity - rather like the domain builder. They want to start a default "kboot" kernel and have that display the user interface, offer network boot options, etc, and then kexec to the "real" kernel. Their requirements are similar to ours, so kboot_sl is worth a look. I think this is an approach we should look at. It gets us: * Interactive bootloader * Guest kernel stored in its own filesystem * Network boot * Multiple filesystem support (in principle, anything that Linux supports) * Exotic boot support (boot from http, ftp, ssh - good for developers) * A common platform with other software A port of Grub(2?) might also be nice to have, but I suspect that (given a working kexec) kboot_sl will be easier to get working and will be more flexible. Cheers, Mark On Wednesday 12 July 2006 15:44, Ian Pratt wrote:> Folks, > > Please find attached a proposal for the Xen development roadmap for the > next 18 months or so. > > The roadmap gives a "state of the nation" summary and provides > suggestions as to priorities going forward. I''ve tried to be inclusive > and collect together many of the various features that folks have asked > for, along with some implementation ideas to guide developers. The > document ends with a table that has a suggested timeline for landing > some of the more critical features. [NB: The doc was written over the > period of a couple of months, so a few of the work items mentioned are > already now complete and checked-in.] > > Lets discuss the proposal on xen-devel and at the xen mini summit, then > get it tabulated onto the wiki as a live document. > > Best, > Ian > > [latex text version follows, PDF attached] > > \title{The Xen Roadmap} > \author{Ian Pratt, July 2006} > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > \maketitle > > > Since around October 2005 the main focus of the Xen project has really > been around stability: getting the 3.0.0 release out, and then ongoing > hardening in preparation for FC5 and SLES10. We''re now in the > fortunate position of having really very few outstanding bugs, and now > have excellent automated test infrastructure to help us maintain this > level of quality. > > It''s now time to turn thoughts to further development and work out > where we want to take the project toward the next major release, Xen > 3.1 (or 4.0, whatever we end up calling it). This document lists what > I believe are the priorities over the coming months, and hence what > the core Xen team will be investing effort in. > > \section{Releases} > %%%%%%%%%%%%%%%%%% > > Rather than the `big bang'' 12 month development cycle used in Xen 1.0, > 2.0 and 3.0, we want to make the development toward the next major > release far more incremental. We aim to continue the current practise > of going through a stabilization phase and having point releases every > 10-12 weeks. Just as with Linux, the point release will be maintained > with bug fixes until the next point release. > > One side effect of this approach is that at the time of a release not > all features that are new in the code base may necessarily be stable > in time for the release. We won''t hold the release, but will simply > disable such features or document the issues. The aim is obviously to > avoid regressions in functionality that worked in earlier releases. > > \section{Performance, Scalability} > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > During the Xen 3 development cycle the vast majority of effort went in > to correctness rather than performance and scalability. It''s pretty > clear from looking at historic benchmark data that some performance > bugs have crept in along the way: there really is no good reason why > xen3 should perform worse than xen2 on any benchmark, but measurements > indicate there are currently regressions on some benchmarks. Clearly, > investigation is called for. I expect there to be quite a bit of low > hanging fruit, in particular it is likely that various of the Linux > kernel version upgrades have broken certain assumptions causing us to > exercise `slow paths'' rather than the fast path we expect it to be > using. These need to be tracked down and fixed. > > We''re actually quite well-armed with performance monitoring tools these > days, so hopefully this shouldn''t be too difficult: xen-oprofile is a > sample-based profile system useful for looking at system-wide CPU > consumption. The xen software performance counters are useful for > tracking event occurrences in Xen (and hence spotting anomalously high > counts of supposedly rare events), and can also be used to collect > various histograms of scheduler and memory management data. There is > also xentrace which can be enabled to collect timestamped trace > records into a buffer to enable detailed event time-lines to be > collected. > > We expect the vast majority of Xen deployments to be on two socket > server boxes, with some four socket, and we expect dual core to be > commonplace. Hence, we believe that the sweet spot to optimize xen for > is 1-8 CPU systems. 32 way and larger systems are currently supported > and getting good performance on them is certainly desirable in the > mid-term. However, we must endeavour to ensure that such optimizations > do not harm smaller system performance. If necessary, compile time > target selection could be used, but I hope this won''t be required. One > of the key features which will help both small and large system > performance is support for NUMA-aware memory allocation. This is > useful even on two socket AMD boxes. I expect to see the core Xen > mechanisms in the tree in short order, but implementing page migration > (particularly the policy for doing so) will be a longer term goal. > NUMA topology-aware CPU scheduling will also need to be developed, and > is discussed in a later section. > > \section{Guest API stability} > %%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > The Xen 3 Guest API is intended to be a stable interface that will be > maintained in a backward compatible fashion. Old 3.0 guests should run > on a new hypervisor. The converse of running new guests on an old > hypervisor has not been a commitment, but we should obviously plan to > transition toward this. This becomes more important as soon as a given > version of Xen starts gaining wide use in an Enterprise Linux distro > (e.g. SLES10). > > The xen guest API includes all the virtual IO interfaces > (e.g. netfront, blkfront etc). We expect there to be some evolution > in these protocols, but nothing that can''t be supported in a > compatible fashion: xenbus provides the necessary mechanisms for > enabling feature selection, or even selection of alternate front or > backend drivers. > > So far, there has been no commitment to maintain a stable interface > between privileged domains (dom0) and xen, or between xend and > dom0. We currently expect xen, the dom0 kernel, and the xend tool > stack to be a ``matched set''''. These interfaces will continue to evolve > for at least the next six months, so no stability is guaranteed, > though interface breakage will be avoided where possible as it''s > clearly inconvenient for developers. Interface version numbers should be > updated when this happens to avoid subtle incompatibilities. > > The continued evolution of the privileged domain interfaces is being > driven by a number of factors: changes such as the new pci device > pass-through code, support for IOMMUs, integration of ia64 and ppc, > more fine-grained security capability delegation. It should be a long > term goal to stabilize this interface, but for the moment 2.6 Linux is > in the favoured position of being the in-tree privileged domain OS, > with NetBSD and Solaris playing catchup. > > One of the key APIs we expect to evolve and then stabilize quickly in > the next 2-3 months is the xen control API, encompassing all the > various XML parameters that configure and control guest domains. The > roadmap for the xen control stack is set out later in this document. > > \section{OS Support} > %%%%%%%%%%%%%%%%%%%% > > The following OS kernels have been ported to the Xen3 32b guest ABI: > Linux 2.6.16, 2.6.9-34.EL, 2.4.21-40.EL, 2.6.5-7.252, NetBSD3, > FreeBSD7.0 and OpenSolaris10. Work is planned to update the Plan9 port > to xen 3, and the API stability will hopefully encourage other OSes to > be ported too, such as OpenBSD. The only x86\_64 OS port is currently > 2.6.16, but other ports are underway. > > Since the 3.0.0 release there have been various backward compatible > enhancements to the Xen guest API, such as the `feature flags'' and > `transfer page''. These mechanisms are intended to make backward > compatibility easier to maintain in future, and will be used to enable > features such as supporting 32b guests on a 64b hypervisor, running > xen kernels on bare metal, and running xen kernels as HVM (fully > virtualized) guests. We would urge maintainers of kernel ports to > adopt the new feature flags and transfer page ASAP if they haven''t > already done so. > > > \section{Getting xen support in kernel.org Linux} > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > The work to get xen support into upstream kernel.org Linux is a > crucial work item for the xen community. Although it costs > considerable man effort right now, it should reduce the maintenance > burden in the long run as hypervisor support will be a more visible > consideration for Linux developers, and hopefully treated as a first > class citizen. > > The full xen support for Linux patch maintained in the main xen > repository is really quite large. As well as supporting Linux as an > optimized SMP guest, it has all the functionality necessary to run > Linux as a domain 0 with access to physical hardware, support for IO > virtualization backends, support for the control tools etc. > > It is clear that getting such a large patch into kernel.org in one go > is infeasible, so a more incremental approach has been adopted. Our > initial aim is to submit patches that enable Linux to run as a simple > Xen guest, without various of the more invasive paravirtualization > optimizations that provide rather better virtual memory and SMP > performance. A patchset has been prepared by Christian Limpach, > Chris Wright and Jeremy Fitzhardinge, and is being iterated on LKML. > > The path into kernel.org has been muddied by the discussion around > VMware''s VMI proposal. VMI proposes an abstraction layer for the > hypervisor to guest API used for CPU and memory management > virtualization, and doesn''t attempt to address the virtual IO, > hardware access, and control tool APIs that the Xen patches > address. VMI is also currently only addressing 32b x86 (Xen is > currently x86\_64 and ia64 too). > > Comparing VMI with a subset of the xen patch, there''s actually quite a > lot in common. The bulk of both patches refactor i386 code to provide > hooks, and it is hoped that these common changes can be upstreamed > quickly, reducing the size of the external patch that must be > maintained. Rusty Russell is taking the lead on this. At some point > there will be a discussion about what the correct hypervisor API > abstraction is, and hence the extent to which paravirtualization can > be exploited to improve performance. The Xen team will be arguing > vigorously that a rich interface that is part of the kernel source > (and hence GPL) is more easily maintainable, and enables `deeper'' > paravirtualization, and hence more opportunities for performance > enhancements. > > It is conceivable that what ends up being merged into kernel.org may > require some changes to the xen 3 guest ABI. We would certainly hope > that these could be made backwards compatible with existing xen guest > OSes, but we may have no choice other than to rev to a Xen 4 ABI. This > would be a great shame as ABI stability is a key hypervisor > requirement, but this would at least be a one off change. > > \section{Control tool stack} > %%%%%%%%%%%%%%%%%%%%%%%%%%%% > > Following on from the discussions at the last Xen summit, a number of > significant changes are planned for the xen control tool stack over > the next few months. In the mid-term, the DMTF CIM model for VM > life-cycle management from the virtualization and partitioning working > group is likely to emerge as the standard for configuring and managing > VMs. Many of the folks in the Xen community are already working in the > DMTF in support of the CIM model, as well as Microsoft in their > Carmine tools. The joint IBM/Novell/XenSource project to create CIM > providers for Xen is of great strategic importance. However, > management via CIM is quite heavyweight and intricate for some > scenarios, so also having a simpler management API makes > sense. Indeed, having such an API will be useful for building the CIM > providers themselves. > > {\em XML config file and conversion tools.} The first stage of > xend development will be to switch to using XML for all configuration > data. We are in the process of drafting a specification for an XML > data-model for storing all VM configuration data, and will be > circulating this on xen-devel/xen-cim shortly. This scheme is > `inspired by'' the CIM data model, but the hierarchy is somewhat > flattened and simplified to reflect xen''s requirements and provide an > easy to navigate model. The intention is that XML config files > conforming to this schema would replace the current python and SXP > xend config files. Since this is a user-visible change, creation of > migration tools will be required. Config files would no longer live > under /etc, but would be loaded into xend when a VM is created, and > then stored as plain XML files under /var. > > {\em xend VM life cycle management and storage extensions.} Another > addition will be implementation of some simple VM life-cycle > management inside xend. The current tools already have a very limited > form of this implemented by the xendomains script which will preserve > VMs across host reboots using save/restore. Adding VM life-cycle > management to xend means that we will store state for VMs even if they > are not currently running. This will enable us to make operations like > VM save/restore less dangerous from a user point of view by tracking > the resources (in particular, disk images) they have reserved even > when they are suspended. Further, support for simple storage > management will be added to xend, enabling call-outs to create new > disk images, create CoW snapshots etc. These will be implemented in > scripts that provide the functionality for different storage backends > e.g. LVM, file based, qcow, etc. > > In previous and current Xen releases the protocol between `xm'' and > `xend'' has not been documented and has undergone rapid > change. Although the internal xmlib API has generally remained > relatively stable this has not been a popular interface for developers > building on top of Xen -- most have resorted to using `expect'' scripts > to drive the xm command line. > > {\em XML-RPC xen control API plus C/C++/perl/python bindings.} It''s > now important that we define a suitable message protocol and > associated API, rapidly switch xend and xm over to using it, and then > provide client bindings for common languages > e.g. C++/perl/python. Libvirt will hopefully help fulfil this > role. Draft specifications for the message protocol are being prepared > and will be circulated shortly. The scheme is xml-rpc based, using an > SSL secured https transport to enable secure remote management, or > unix domain sockets for lightweight local management. > > The protocol will support a notion of logging in as a given user, > using PAM to authenticate on the server. Since some of the RPC > commands may be long running (e.g. a VM relocation), the protocol > supports > the notion of tasks running asynchronously in the background. The > client can poll for their completion or list outstanding tasks. > > In addition to securing the control protocol, we also need to secure > the networking connections used for VM relocation. The intention is to > split the VM relocation operation into two, issuing a relocate\_receive > that generates a token that then must be presented when initiating a > relocate\_send. Since these network connections are performance > critical, it is important we retain the option to have just > authentication without mandating encryption. > > A further area where work is clearly required is developing a decent > web GUI for Xen. There have been a number of previous attempts at this > by various folk, but none have really gained traction. We would > envisage that the GUI would be implemented using presentation layer > code that would communicate with xend via the control protocol and > generate HTML/javascript to return to the browser. Possibly picking > one of the existing web UI''s and trying to get it in-tree will help > focus efforts. > > > \section{Virtual Hard Disk Images} > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > Xen 3.0 supports a wide range of storage options for guest virtual > disks, including physical partitions (LUNs), LVM volumes, and `loop'' > files. Use of loop files has proved popular as they support sparse > allocation, and are very easy from a management/backup point of view > (e.g. you can just copy the files around). However, the loop driver > has some serious deficiencies, in particular it buffers writes very > aggressively, which can affect guest filesystem correctness in the > event of a host crash, and can even cause out-of-memory kernel crashes > in domain0 under heavy write load. Further, using sparse files > requires care to ensure the sparseness is preserved when copying, and > there is no header in which metadata relating back to the VM can be > stored. > > Given the popularity of the file-backed model, providing a robust and > high performance solution that supports it is highly > desirable. Rather than using raw image files, it makes sense to move > to a format that supports header metadata, sparse allocation > independent of the underlying filesystem, and copy-on-write > support. We have spent time evaluating a number of existing formats: > VMware VMDK, Microsoft VHD, and QEMU QCOW. VMDK is a hodgepodge union > of several different formats used by VMware, and doesn''t have much to > recommend it. Further, the licencing terms aren''t entirely clear about > its GPL compatibility. Microsoft''s VHD format is pretty nice, but > explicitly doesn''t allow open source implementations. QEMU QCOW looks > to be the best of the bunch: It''s been part of the QEMU stable for > over 18 months and had quite widespread use, and supports advanced > options like compression and AES encryption as well as sparse > allocation and copy-on-write. > > The world doesn''t need another virtual hard disk format, so with > Fabrice Bellard''s blessing (QEMU author) we''re strongly advocating > that the Xen project adopt it. The licence on the current QCOW > implementation is BSD, which makes implementation with Xen and even 3rd > party closed-source tools easy (e.g. Virtual-to-Physical transfer > utilities). > > Given the various problems with the `loop'' driver, this doesn''t seem a > good starting point for implementing qcow support. The easiest > approach seems to be to build on the `blktap'' approach that is already > in the tree and provides a way of implementing virtual block devices > in user space. Work is well under way to implement a `ublkback'' driver > that supports all of the various qemu file format plugins. A special > high-performance qcow plugin is also under development, that supports > better metadata caching, asynchronous IO, and allows request > reordering with appropriate safety barriers to enforce correctness. It > remains both forward and backward compatible with existing qcow disk > images, but makes adjustments to qemu''s default allocation policy when > creating new disks such as to optimize performance. > > > > \section{Resource Control} > %%%%%%%%%%%%%%%%%%%%%%%%%% > > One of the big deficiencies with Xen 3.0.2 is the static assignment of > VCPUs to single physical CPUs. Although domain creation attempts to do > some crude load-balancing placement on physical CPUs the assignment > is not updated, which can lead to gross imbalance between CPUs if > e.g. all even numbered VMs were to exit. Currently manual intervention > with `xm vcpu-pin'' is required to rebalance things. > > A new CPU scheduler has recently been completed which should solve these > issues. It supports automatic migration of VCPUs between their > allowable CPU set, and actively balances VCPUs across CPUs > in an attempt to maximise throughput. > > The scheduler supports a notion of `weighted fair share'', enabling the > relative weights of guests to be set e.g. this guest should be able to > get twice as much CPU as this guest (assuming they are both CPU bound > and runnable). Further percentage CPU ceilings may be set to constrain > the > consumption of a guest, e.g. limiting it to 10\% of a CPU even if the > CPU is otherwise idle. [This non-work conserving option is useful in > hosting environments where it is sometimes desirable to stop customers > having the opportunity to get used to more resource than they''re > paying for.] > > For multi VCPU guests the scheduler tries to ensure that each > (runnable) VCPU accrues CPU time in roughly equal fashion. This avoids > bad interactions with the guest OSes internal scheduler. Currently, no > attempt is made to gang schedule VCPUs belonging to the same > guest. For most workloads this seems to work OK. In future we may have > to investigate schemes that switch to gang scheduling under certain > circumstances, or otherwise use bad pre-emption avoidance (e.g. don''t > pre-empt while kernel locks held) or bad pre-emption mitigation > strategies (e.g. directed yield). It may be possible to dynamically > spot groups of VCPUs that are actively communicating and gang schedule > them. We will need to extend the scheduler to better understand CPU > topology and make informed scheduling decisions in the presence of > hyperthreading or NUMA architectures (in the short term, we can use > CPU affinity settings to assist this). Due to potential information > leakage through cache timing attacks between domains sharing the same > hyperthreaded CPU core, we may need to ensure that hyperthreads > belonging to the same core are gang scheduled for the same domain (of > shared with a trusted IO domain). > > > One other scheduling feature that has been discussed is instrumenting > domain 0 to enable the work that it performs on behalf of other guests > to be accounted to those guests. The simplest way to do this is just > to do some simple counting of grant transfer and grant map events for > each domain, and then bill the CPU time consumed by domain 0 in > proportion (with an adjustment factor to account for the CPU > differences between disk and network IO). Although simple, this scheme > would probably yield most of the benefit, and avoid CPU-bound domains > being penalized in the presence of bulk IO. > > > Support for network QoS is in pretty good shape: The netback driver > can implement token bucket rate limiting (``x KB every y > microseconds)'''') on any virtual interface to enforce simple max rate > control. More complex queueing and scheduling strategies can be > implemented simply by invoking Linux''s existing iptables and > traffic\_control facilities. > > Some control over disk IO bandwidth is now possible using Linux''s > inbuilt CFQ IO scheduler. Using `ionice'' it is possible to set the > relative priorities for disk accesses. More experimentation is > required to determine whether this will prove to be a sufficient means > of control, or whether some further controls (e.g. implemented in > blkback) will be necessary to provide xen admins with the features the > require. Disk scheduling is a well-known thorny research problem, and > best driven by user requirements. > > > > \section{HVM (fully virtualized) Guests} > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > As CPUs with Intel VT and AMD-V support come to market in volume, > ``HVM'''' guest support is now a critically important core feature of Xen. > The current support we have today is ``OK'''', but we know we can do a lot > better, and really make the new hardware fly. This involves > substantial surgery to a number of key subsystems, but we seem to be > making good progress. > > \subsection{Shadow Pagetables} > > One of the most important and complex subsystems in Xen is the shadow > pagetable code. For paravirtualized guests, this is typically only > used when guests are undergoing live relocation, but for HVM guests it > is turned on the whole time and is critical to performance. The code > has to support a number of different modes of operation, and deal with > the differences between guest and host page table levels -- the code > supports 2-on-2/3/4, 3-on-3/4, and 4-on-4. > > The current shadow pagetable implementation is large and very complex, > partly as a result of having been hacked on by a lot of different > people each addressing their own requirements. The algorithm itself > isn''t too bad, but we''ve now accumulated a lot of profile data from > different OSes to enable us to do a better job this time round. We > have designed a new algorithm and have embarked on a complete rewrite > of the code, which will take some time complete. Because of the shadow > pagetable code''s importance, testing it is a major task as the test > matrix of different OS versions and configurations is vast. We will > need to run the two code bases in parallel for some period of time, > either in different trees, or possibly in a single tree with a boot > option. We have a list of optimizations and heuristics we intend to > add to the new implementation once the core is stable. In the > meantime, there are various folks continuing to fix and optimize the > code currently in the xen tree, which further helps inform the new > design. > > \subsection{QEMU} > > QEMU has proved to be very helpful for providing Xen HVM guests with > emulated IO devices. However, Xen''s current ``qemu-dm'''' code has > diverged quite heavily from mainline qemu, which means that we can''t > as easily capitalise on enhancements made to mainline qemu, and also > makes it harder for us to contribute enhancements back to > Fabrice. Fixing this is a priority. We are developing a > re-implementation of qemu-dm that is maintained as a patch queue > against an unmodified snapshot of mainline qemu, and hope to have this > ready for testing soon. Like the shadow pagetable code, this is going > to require extensive testing even prior to inclusion in -unstable. > > Catching up with the latest version of qemu has a number of nice side > effects. It gets us Anthony Liguori''s improved VNC server code for the > framebuffer, removing our dependency on the rather obtuse libvncserver > library. Further, it gets us support for emulated USB devices, most > interesting of which is a USB mouse. The USB mouse protocol supports a > mode of operation whereby it provides absolute x,y co-ordinates > rather relative motion events (which the OS usually turns into > absolute co-ordinates using some unknown `black box'' algorithm that > has mouse speed and acceleration parameters). Being able to inject > absolute co-ordinates means that it is easy to arrange for our the > HVM guest cursor to perfectly track the local cursor in the VNC > viewer, regardless of the user''s mouse settings. > > Although not immediately on the horizon, there are plans to give qemu > a slightly extended role in Xen. The `V2E'' research work in Cambridge > has shown that it is possible to move the execution state of a guest > from a Xen virtual machine into qemu and back out again. In the > context of the research work the aim was to enable high-performance > taint tracking to be implemented, where the guest ran at full speed as > a xen guest until it accessed a tainted value at which point execution > was moved on to the qemu emulator which was able to monitor execution > closely. At some point later execution would be moved back to Xen. As > pointed out by Leendert, a similar approach can be used for > transitioning into IO emulation, cleaning up the current interface and > possibly providing performance optimizations when many IO operations > are performed in close proximity. This technique also solves a thorny > issue with Intel VT systems, which unlike AMD-V do not provide h/w > assistance for virtualizing the legacy x86 `real mode''. Today, we have > the `vmxassist'' code containing a crude emulator to try and handle > this case, but the code fails in the presence of certain complex 16b > applications, such as MSDOS or the SuSE graphical boot loader. Having > the ability to throw execution onto QEMU which has a complete real > mode emulation would solve this problem. > > \subsection{Paravirtualization Enhancements} > > Although we can run completely unmodified guest OSes in HVM mode, > there are significant performance advantages that can be achieved by > selectively adding paravirtualization extensions to guests. The nice > thing about this model is that the OS author can boot their OS > unmodified, then incrementally enhance the OS to exploit Xen''s > paravirtualization hypercall API to improve performance, for example, > adding paravirtualized IO drivers, then paravirtualized virtual memory > and CPU operations etc. > > The Xen API''s hypercall transfer page assists in allowing the precise > method used to make the hypercall to be abstracted from the guest > (e.g. INT82 in the full-paravirtualized case, VMCALL on VT, VMMCALL on > AMD-V). However, the exact method for installing the page (e.g. by > writing MSRs) and passing in-memory arguments to hypercalls has yet to > be finalized, though there are prototype patches in existence. Making a > decision on which approach to checkin is a priority. > > Typically, the paravirtualizing extension that has most impact is > switching from using IO emulation to using PV drivers. This can be > done by taking the core of the existing netfront/blkfront/xenbus > drivers and providing appropriate wrappers to enable them to be built > against a native (non Xen) Linux tree, and thus loaded as modules into > the native kernel running as a HVM guest. PV drivers can similarly be > prepared for other OSes. > > \subsection{Save/Restore/Relocation} > > One of the key features missing in Xen''s current support for HVM > guests is save/restore. Several subsystems need to be updated to add > this support. We need to exploit qemu''s ability to `pickle'' the IO > state of a guest and be able to pass it down a file descriptor. We > need to add code to Xen to enable the state of all of the > high-performance emulated devices such as the PIC, APIC, IOAPIC etc, > to be read out via a the get\_domaininfo dom0 op. We then need a simple > update to the xc\_save code to work with auto-translate shadow mode > guests. The restore operation requires very similar changes, enabling > device state to be `unpickled'' in both Xen and qemu. Having got > save/restore working, live relocation should actually be relatively > straight forward, just enabling `log-dirty'' mode for auto-translate > mode guests. > > \subsection{SMP Guests} > > The current HVM code only stably supports uniprocessor guests, though > work is underway to robustify SMP guest support. Xen already contains > APIC and IOAPIC emulation modules, but this code along with the > existing shadow pagetable code lacks synchronization in certain > places. The new shadow mode code should go a long way to solving > this. Work is also ongoing to add ACPI BIOS support, hence allowing > the emulated platform to look like a modern PC. The first aim for > supporting SMP guests is correctness, and once we have that focus on > performance and scalability. Getting fair performance for two and four > way guests should be achievable for many workloads, but this is an > area where paravirtualization really helps. Eventually, it would be > good to support CPU hotplug emulation in Xen. > > \subsection{Mid-term goals} > > Looking toward the mid-term, there are a number of projects around HVM > guests that would be good to see implemented. > > One of the key desires is to move the qemu device emulation out of > domain 0, and run it in a `stub domain'' associated with its HVM > guest. A `stub domain'' is the effectively the execution context > associated with a domain that a paravirtualized guest would normally > use, but is currently unused for HVM guests. Belonging to the same > domain, any CPU time spent executing in the stub domain is naturally > accounted to the guest. Like normal paravirtualized guests, stub > domains are strongly isolated from other domains. However, given the > close relationship with the HVM guest, executing transitions between > the two occur faster than transitions to a secondary domain. Further, > belonging to the same domain means that the stub domain can easily map > memory belonging to the HVM guest. These properties mean that stub > domains are an ideal place to run qemu, providing improved performance, > accurate resource accounting, and surer isolation. > > As a user space application, Qemu can''t run in the stub domain > directly, but requires an operating system kernel. The neatest way of > doing this would be to link qemu against `minios'', which is > effectively a library operating system for just this purpose. Since > minios makes use of a broad range of libc calls, it is likely that > minios will take some time to reach the required level of support. In > the meantime, we can just use a xen linux kernel, with a minimal > config to keep the size down. Since protection between user space and > the kernel is irrelevant in the context of running qemu as the sole > application, we could optimize performance by running user-space at > the same privilege level as the kernel, effectively turning system > calls into plain jmp instructions into the kernel followed by a ret to > return. > > An interesting thing becomes possible once we have qemu running in > stub domains and interfacing with the HVM guest via the `V2E'' approach > described previously: it becomes quite easy to enable unmodified > guests to be run on CPUs that don''t support VT or AMD-V. Gust > execution in user space would proceed in the normal xen fashion, but > any transition into the kernel would result in the guest being > transferred on to qemu for emulation, which would then transition back > to native execution when the guest exited the kernel. Having the > emulation running in a stub domain is clearly important to allow the > resource spend in emulation to be correctly accounted. Performance > would clearly not be as good with VT/AMD-V approach, but this does > provide a good way for folks with older hardware to experience > unmodified guests on Xen. > > >From our point of view, QEMU''s biggest failing is that the devices it > > emulates are quite old, and lack some facilities that could > potentially lead to better performance. It would be nice to have an > emulation of a SCSI HBA, as most OSes typically treat these > differently from an IDE device and make more use of the ability to > have multiple outstanding requests, which is essential for good IO > performance in a virtualized environment. BOCHS has a simple SCSI HBA > emulation, and it may be possible to use this as a starting point. > > Emulating a different network device would also offer benefits. A > network device that supported checksum offload, jumbo frames, or TSO > (Transmit Segmentation Offload) could all offer CPU savings in the > guests. Since IO performance is typically dominated by the number of > `vmexit'' operations required, care is required when selecting which > hardware device to emulate to ensure that the corresponding driver > will generate a minimal number of exits. Older hardware is sometimes > better in this respect, since the driver writer knew certain io port > and mmio operations were likely expensive, and thus went out of their > way to avoided doing them. Since such io port and mmio operations > typically generate vmexits in the Xen case, minimizing them is helpful > for performance. > > > \section{Paravirtualized Guests} > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > As stated earlier PV guest support in Xen looks in good shape from a > stability point of view, but we do need to pay some close attention to > benchmarking and tuning, particularly for large SMP guests. There is > undoubtedly plenty of low hanging fruit that can be addressed to > improve Xen performance -- it just hasn''t been a priority given the > state of the competition. > > One area that could certainly do with some attention is the 64b Linux > guest ports. The code could do with a full code review, specifically > to remove some of the unnecessary divergence from the i386 xen code, > and to remove some of the modifications relative to native that are > now unnecessary. There are a number of opportunities for investigating > optimizations too. The current code was written assuming that the ``TLB > flush filter'''' found on AMD Opteron processors would become > ubiquitous. Sadly, Intel haven''t adopted this, and AMD have dropped > the feature from future chips. There are a number of places where TLB > flushes occur in the current code where it was expected that the flush > filter would actually annul them. Given that we can no longer rely on > this, some modifications to the virtual memory virtualization > implementation is called for to reduce the number of TLB flushes. One > particular idea that has been mooted is using the global bit in PTEs > to preserve both Xen and User mappings across system calls. This needs > to be implemented and benchmarked. Hopefully Intel/AMD will extend the > x86 architecture with an address-space tagged TLB implementation at > some point, providing a clean solution. > > One area that hasn''t received much attention recently is the ``live > relocation'''' feature since the 32b PAE and 64b hypervisor variants were > introduced. Although the current code seems to work, it''s never been > `productized'' on these hypervisor variants, so shouldn''t be relied > upon. The new shadow pagetable code being developed primarily for > improved HVM guest support should also help address this concern, > greatly simplifying the current code. > > We also need to do some work on the xc\_restore function itself, > modifying it to support lazy allocation of memory to avoid the current > issue whereby when relocating a guest you (temporarily) need to > allocate enough memory for the maximum memory size of the guest even > if its current size is much smaller due to memory ballooning. Further, > we could probably improve guest down-time by a few 10''s of > milliseconds if we did some streamlining of the various hotplug > scripts and python code that do re-plumbing of network and block > devices after a live relocation. > > > \subsection{PV Block IO} > > The current block device protocols and implementations are in pretty > good shape. The work to add a secondary backend implementation called > `backtap'' to support high-performance file-based qcow implementation > has previously been discussed. This blkfront implementation uses the > same guest IO API, and can be used interchangeably with the current > blkback and blktap backends. > > One area that does need some discussion is whether the protocol should > be extended to support in-band metadata operations, for example, > enabling the front end to issue something akin to ioctl on the > backend. Such requests could probably be implemented fairly straight > forwardly, as a special variant of a write operation that also returns > data in-place. We would need to have well defined meanings and > enumeration of these ioctl-like operations since there may be entirely > different operating system kernels at either end of the block protocol > device channel. However, for `occasional'' ioctl operations it might be > best just to use the xenbus control channel between front and > backend. This can also be used for passing messages the other way. In > particular, we should use this protocol to handle removable media > change events better than we currently do. This mechanism could also > be used to provide notification of virtual disk size changes, enabling > an on-line resize tool to be invoked to provide seamless growing of > virtual disks. > > One extension that has been requested by at least one proprietary > filesystem vendor is the ability to return metadata with both read and > write responses (rather than just noting completion). This meta data > is interpreted by the file system and used to optimize certain > journaling operations. These extensions are currently considered > rather specialized an probably mandate a separate blk protocol. > > Another idea that has been mooted is supporting a scsi-level device > channel protocol, and hence scsifront/back drivers. This would be > useful for controlling more `exotic'' devices that need a richer set of > operations than just read/write/barrier, e.g. a tape drive. It does > introduce a whole load of fairly pointless scsi command creation and > parsing at either end for the normal read/write case, which may end up > resulting in measurable overhead. Experimentation is required. > > > \subsection{PV Network IO} > > One key area that needs some performance tuning is the paravirtualized > network device driver. The current drivers use a page-granularity > protection mechanism to provide good containment and thus minimal > trust between the frontend and backend drivers, but this does exercise > the page protection mechanisms pretty hard. > > There are a number of fairly obvious extensions to the network device > channel protocol that should yield useful performance improvements. > Some of these could be implemented in a backward compatible fashion by > using spare fields in the current protocol, but at some point we''re > likely to want to define the net device channel v2 protocol. This is > not really a big deal as the xenbus control plane is there to ensure > that we get the right drivers bound at each end of a device > channel. > > One cleanup that is required is to modify the way that the checksum > offload protocol is implemented, allowing the offset of the checksum > in the packet to be specified. This should clean up some of the > current fragility we have in domain 0 when checksum offload is enabled > and certain less common protocols are used. > > Another useful improvement to the current code would be to add > a flag to the data area of skb''s that have been the subject > of decrypt-in-place operations (e.g. for IPSEC VPN''s), and only > selectively scrub the flagged pages when adding buffers to the free > queue. > > The current driver uses page flipping as the only mechanism for > transferring data over the device channel. The device channel > mechanism is actually quite flexible, and with a simple bit of > refactoring, it should be possible to give the hypervisor the option > of either copying the packet or flipping the page, making the decision > based on the operation size. The copy operation may become > particularly attractive if high-performance multi-threaded copy > engines start becoming integrated in to northbridges, as has been > speculated in some quarters. > > Another possibility worth investigating is the use of a semi-static > shared memory buffer between the front and backend i.e. the grant > table mappings are set up in advance for the buffer and > left in place rather than being cycled. On the receive path, netback > could directly copy packet payloads into the shared buffer from the > hardware receive buffer, and then netfront could directly copy > out of the shared buffer and into a local skb (2 copies, but no > hypercalls). > > It would be nice to eliminate the copy in netfront, but this is quite > hard as the skb could end up queued for an arbitrary amount of time in > an application socket buffer, hence consuming resource in the shared > buffer. We could grow the shared buffer if we run out of space, but > there''s no real bound to the size that would be needed and the number > of in-flight buffers we''d have to track. Possibly some adaptive scheme > that uses copying for buffers that are likely to be long lived or when > we are low on shared buffer resource would be possible. However, the > downside of this approach is that guests have to invest more trust in > their backend domains as they retain writable mappings to the packets > sitting in kernel buffers. A buggy or malicious backend could modify > the contents of a packet buffer after the guest kernel had validated it, > and could very likely cause it to crash. > > There are also performance enhancements to be achieved through > supporting some of the higher-level features provided by modern server > adaptors. Jumbo frame support would clearly be useful, both in domain > 0 and from guest VMs, but many Ethernet installations still use a 1500 > byte MTU, so couldn''t benefit. TCP Segmentation Offload (TSO) for the > transmit path would be useful, whereby the frontend passes the backend > a very large `super packet'' for it to segment and transmit (ideally, > passing it to the NIC to do the segmentation if we can get the super > packet through the bridge and routing code). Even if we end up having > to do the segmentation in the backend we''re still better off than we > were passing multiple packets as we''ve reduced overhead and made it > easier to batch work across the device channel. > > To support TSO, we need to extend the device channel descriptor format > to flag packets that should be segmented. Since super packets are > unlikely to be physically contiguous, we also need to be able to > support data fragments in the descriptor format. This should be a > relatively simple extension, adding chaining to netring descriptor > entries. > > In future, we should consider how we might export byte stream (TCP) > data between domains in a high performance fashion. This would be > particularly useful between VMs running the same machine, or when TCP > Offload Engines (TOE) in hardware NICs become commonplace. Similarly we > should have an extended interface for exporting RDMA (Remote DMA) > capabilities. > > Examining oprofile traces of Xen systems under heavy network load it > looks like the network bridge code running in domain 0 takes a > surprisingly large slice of the CPU. It''s quite possible that we''d do > better with some cut-down streamlined bridge code that just handles > the common case that typical Xen installations use it under. At the > very least, we need to do some investigation into why the current > bridge code shows up in profile results as high up the ranking as it > currently does. > > The current netfront/netback approach is geared toward having domain 0 > (or another domain) acting as the switch/router for packets between > other domains. There are some circumstances where it may be more > efficient to create virtual `point-to-point'' links between VMs, > enabling them to communicate directly without going via a software > switch/router. This would be akin to connecting two netfront devices > together, allowing very high performance networking between two VMs on > the same machine. This could be used in applications where there is > effectively a processing pipeline of data being passed between > VMs. Were one of the VMs in the pipeline to be migrated to a different > machine, the netfront on each end of the point-to-point link could > revert back to being connected to a netback driver, fulfilling the > connection via the external network, albeit at lower bandwidth. > > While adding all these extensions to the protocol we should remember > that these same paravirtualized drivers are used to provide > paravirtualized IO within HVM guests, and ensure that they work in > both scenarios. > > > \subsection{Client Devices} > > Although most Xen deployments are currently targeted at server > machines, improving support for desktop and laptop usage is clearly > important, not least to encourage developers to run xen on their main > machines. Having excellent support for client devices such as USB and > the frame buffer is clearly required, along with good power management. > > Way back in the days of Xen 2 we had support for passing control of > individual USB devices over to a guest VM. However, although > apparently stable, the implementation wasn''t ideal and was never > updated to Linux 2.6. The best you can do today on Xen 3 is to assign > a whole USB host controller to a guest using PCI pass-through, rather > than individual hub ports as before. > > There have been patches floated for adding this support back to Xen 3, > but there''s never been quite the impetus to get consensus and get > something checked in. The situation has recently become more > complicated with the USB-over-IP code now appearing in Andrew > Morton''s Linux tree, perhaps suggesting that the design should be > revisited.The TCP transport would be replaced with a reliable > byte-stream device channel transport to avoid the need for network > configuration. It would be good to return USB support to Xen 3 as soon > as possible, as this will solve a number of requirements around audio > virtualization, mouse, scanners etc. > > For PV guests, Xen currently only supports a virtual serial > console. If a graphical console is required the guest needs to run its > own networked frame buffer console, typically Xvnc. This is less than > ideal as it requires networking to be working in the guest before the > console can be used. Serial console serves perfectly well for boot, > though some users get confused by the differences between a serial > console and a typical Linux Virtual Terminal (VT). > > The best solution to this is to implement an in-kernel fbdev > paravirtual frame buffer driver, which uses a shared memory device > channel to make the frame buffer available to domain 0, where it can > either be rendered locally, or converted to a network frame buffer > protocol. This is precisely what happens with the console of HVM > guests today, so will also help unify the `look and feel'' between PV > and HVM guests. Having the PV framebuffer as a kernel fbdev means that > it will be able to emulate a text mode and display messages from > fairly early in the boot sequence, again making operation closer to > native. > > In a basic implementation, the framebuffer control software in domain > 0 would run periodically and either copy the shared memory region to the > local display window, or generate the appropriate VNC updates. Scanning > the whole frame buffer is obviously inefficient, particularly as there > are frequently no updates, or the updates are quite localised (e.g. a > flashing cursor). The current HVM frame buffer code implements a > scheme whereby page-level write-protection of the framebuffer is used > to trap updates, and hence provide a (very) rough indication of which > areas of the frame buffer need scanning for updates. Since this code > primarily uses VNC as a backend, the current framebuffer contents is > compared against a snapshot to enable a more compact network encoding. > The PV framebuffer should ideally share much of this code. Rather than > using a page-fault based approach, if the guest maps the framebuffer > once from a single set of pagetable pages, it may be possible to get > better performance using `dirty'' bits on PTEs rather than taking write > faults. > > Achieving decent 2D graphics performance really requires more help > than can be achieved simply by monitoring page-level granularity > updates. Ideally, the framebuffer backend code would be provided with > accurate bounding box update rectangles, and explicit `copy region'' > and `fill region'' commands (along with a separate hardware cursor > rather than just rendering it into the framebuffer). In the HVM guest > case, this could be achieved by emulating a graphics card which > supported (and whose common OS drivers supported) copy and fill > operations. In the PV guest case, the kernel fbdev interface is not > rich enough to supply this data, so this implies that we will need to > write an Xserver driver module that supplies this data to the backend > via a side-band shared memory interface. Requiring modification of the > Xserver to achieve decent graphics performance is unfortunate, but > there is precedent -- this is the approach VMware have taken. > > Anthony Liguori and Markus Armbruster are working on a PV framebuffer > patch that addresses many of these issues and we hope to commit it > shortly. > > Achieving decent 3D graphics virtualization requires a substantially > higher-level interface than that for 2D. With both the Xserver and > Microsoft window systems moving in the direction of using 3D rendering > even for desktop graphics, 3D graphics is becoming mainstream and not > just the preserve of games and CAD/CAM. We will need to investigate > drivers that encapsulate and transport OpenGL and/or Direct3D commands > into backend domains where they can be rendered by the 3D graphics > hardware. There are already a couple of projects that are > investigating this is the context of OpenGL: Jacob Gorm Hansen''s work > presented at the last Xen summit, and a new project at CMU/Toronto > based on Chromium. > > \subsection{Smart IO devices} > > There is no denying that IO virtualization in software incurs extra > latency and CPU overhead relative to native. For applications that > require the best possible performance, some help from the IO hardware > is required. For over a decade their have been specialist `smart'' > network interfaces available that have had the necessary hardware > support to be accessed directly from user-level applications in a safe > and protected fashion. Most of these interfaces have been targeted at > the High Performance Computing market, designed for doing low-latency > message passing, but some newer interfaces also support standard > TCP/IP/Ethernet protocols. The requirements made of a network device > to be able to provide safe direct access from guest virtual machines > is very similar to those for providing direct access to user-level > applications in a traditional non-virtualized environment. We > anticipate many of these smart network interfaces becoming quite > mainstream over the next couple of years as virtualization becomes > ubiquitous. The same principles can also be applied to storage access, > but in our experience the benefits are less pronounced. > > Work by IBM has already demonstrated direct VM access to Infiniband > hardware on xen, and the code for this is available in the > ext/xen-smartio.hg tree on xenbits. In future, we expect to see > support added for the Level5 and NetXen (previously known as UNM) > smart NICs which are more like traditional Ethernet interfaces. > > Typically, the main device driver is run in domain 0, which is > responsible for allocating the NICs resources to other VMs and hence > allowing them to map page-aligned control regions of the PCI device''s > address space into their own virtual address space. The main driver > also controls what memory pages each of the virtual NICs is allowed to > issue DMA operations to/from. Guest VMs use a small unprivileged > driver to operate the virtual NIC''s free/receive/transmit queues via > their control area. The NIC typically also supports some QoS traffic > shaping of outbound traffic from a virtual NIC, and may support more > sophisticated receive demultiplex and inbound/outbound firewalling > options beyond simple layer-2 demultiplexing. Having such features > implemented in hardware is important as users typically don''t want to > give this up when switching from the current s/w solution. > > Before the code can be included in the main xen tree we need to have > decided the interface through which these smart NICs interact with > Xen''s memory management to ensure that a guest requesting DMAs to a > page is actually the owner of that page. Since most OSes tend to > re-cycle network buffers it is usually the case that network packets > are received into and sent from a relatively small and static pool of > memory. This is quite a simple scenario that can be dealt with by > pre-pinning the buffers with the NIC and with Xen. Storage poses more > of a challenge as the pages used for DMA will be spread throughout the > system, particularly for writes (though writes are less latency > sensitive). Clearly, making decisions on this interface should be > strongly influenced by the support planned for chipset IOMMUs. > > Another area where care needs to be taken is consideration of the > interaction between VM relocation and smart NICs. In general, it is > not safe to save/restore/relocate a VM that has direct access to > hardware. Given that access to the interface is under the control of > the domain 0 main driver, it is typically possible to arrange for this > operation to be safe. However, relocating a VM to another physical > machine would require that the destination machine had the same smart > NIC hardware, which is inconvenient. We are considering the > possibility of having a pluggable driver architecture that could enable > the domain builder to `slide in'' a different low-level driver suitable > for the new machine. > > \subsection{PV Filesystem-level Virtualization} > > Block-level IO virtualization provides an abstraction for accessing > storage that users will be very familiar with. However, a > filesystem-level abstraction offers a number of interesting > possibilities too. This would work rather like a network file system > such as NFS or OpenAFS, but without the need for networking as a > shared memory device channel transport between fsfront/fsback would > employed. Thus, ``xenfs'''' would enable one domain to export a subtree of > its file system to be imported and mounted by another virtual > machine. A range of different semantics could be implemented, > including a typical coherent shared file-space, or copy on write. > > The xenfs approach is quite interesting as it provides a very high > performance mechanism for guests to share data in a coherent fashion > via shared buffer cache mappings to the same file. A shared buffer > cache could also avoid some IO, and yield memory savings. Implementing > this for PV guests requires use of a new type of page fault, > ``copy-to-write'''' which is different from ``copy-on-write'''' in that the > all mappings to the immutable old page must be updated to the new > copy. Fortunately, an implementation of CTW faults for Linux has already > been developed by the embedded systems community, who need it when > mapping pages stored in Flash memory. > > Mark Williamson is working on the XenFS implementation. > > > \section{Core hypervisor} > %%%%%%%%%%%%%%%%%%%%%%%%% > > One of the key features we''d like to add to Xen soon is the ability to > run a mix of 32b and 64b paravirtualized guests on a 64b hypervisor > (just like you can with HVM guests already today). > > Because of the similarity in pagetable formats, it should be possible > to run 32b PAE guests on a 64b hypervisor pretty straightforwardly and > with good performance. We just need to provide a `compat32'' version of > the hypercall table and export a 32b version of the `m2p'' table. Since > the hypervisor can live outside the 32b address space of the guest we > can exploit the code in the Linux port to allow a variable sized > hypervisor hole to enable us set the hole size to zero, giving the > guest the full address space. Supporting non-PAE guests is also > possible, but would require the use of shadow pagetables to convert > between the guest and host pagetable formats. Given the resulting > performance hit, it''s probably best to just stick to PAE guest > kernels. > > Work on adding NUMA support to Xen is important now that integrated > memory controllers on CPUs are commonplace. The goals for this work > have been discussed earlier in this document. > > Right now, Xen has no support for allowing guest kernels to exploit > superpage mappings to access 2/4MB machine-contiguous memory > regions -- we always factor such mappings into multiple potentially > sparse 4KB mappings. This has the potential to impact TLB usage for > some workloads [NB: it is worth noting that both shadow-mode and > direct-mode pagetable implementations mean that it is typically > necessary to protect kernel mappings on 4KB page granularity so it is > typically not possible to allow kernel use of such mappings.] > > Xen''s memory allocator is already capable of managing and allocating > contiguous memory chunks. However, we would need to add some guest > accounting to control the number of multi-page contiguous regions a > guest is allowed to claim, otherwise we have the potential for one > guest to hog all the contiguous chunks to the detriment of > others. Such accounting is useful today even without superpage > support as privileged guests can use machine-contiguous regions for IO > purposes, and it would be good to bound the number of these. > > Adding superpage support for mappings of user-space memory for HVM > guests is pretty straightforward, though it will always be necessary > to be able to factor one of those mappings later if maintaining the > invariants of the shadow pagetable algorithm requires it. Support for > direct-mode paravirtualized guests should also be possible, though is > made slightly more complicated by an annoying quirk of the x86 > pagetable format that means that linear mappings interact poorly with > superpage entries as it is not possible to generate a trap if a super > page PTE is accessed as though it is an L1 entry. Places in Xen where > we access the guest pagetable via a linear mapping will need to be > audited. On a 64b hypervisor there should be fewer of these as using > the direct 1:1 mapping is typically preferable. > > Power management is also a concern for the mid-term. We can certainly > put idle processes into a deeper sleep state than we currently > do. Looking even further out, we could also even adapt the Xen > scheduler to make CPU clock frequency scaling decisions, and > distribute load across processors to minimize power requirements. > Getting whole-system (as opposed to guest) suspend and hibernate > support would be a useful feature for laptop users. This shouldn''t > actually be too hard, to the extent that Linux supports the function > running native on a given system. We will need to add stubs in Xen to > do the final stage power down and following resuscitation. > > > IOMMUs look set to become common on future x86 server platforms, which > offers a number of benefits for Xen. IOMMUs are clearly useful to > ensure protection when assigning an IO device to a VM: without one a > malicious or buggy guest could destablize the system or read data > belonging to other VMs by instructing the device to DMA to/from memory > pages other than those it owns. As well as providing protection, most > IOMMUs also provide translation of DMA addresses. This means that they > can be used to enable HVM guests to be delegated direct access to a > hardware device as well as PV guests. Note that save/resume/relocate > operations are likely not possible on guests with direct hardware > access, unless the hardware and driver has been designed for this > purpose. > > Most IOMMUs typically don''t support the fault-and-fixup style of > operation that is common with CPU memory management: Any lookup > failure is difficult to recover from, and likely involves resetting IO > buses and devices, and likely results in lost IO operations. Xen must > be able to recover from such situations. > > In the case of a HVM guest that is not actively co-operating with Xen, > Xen has to maintain an IOMMU data structure containing the full set of > guest physical pages the guest may wish to instruct the IO device to > access. It must ensure that each of those pages is pinned and hence > it''s ownership (or mmu type) can''t change. Before a page can be > unpinned and released (for example, by the balloon driver), xen must > remove it from the IOMMU pagetable structure, issue a flush or > invalidate, and wait for confirmation the flush has completed. On a > processor supporting nested pagetables, it may be possible to share > the guest-physical to machine frame translation pagetable with the > IOMMU. > > For PV guests (or HVM guests with PV extensions), the guest can be > more actively involved with co-ordinating what pages are accessible > via the IOMMU. The simplest mechanism for doing this would be to use > the hooks provided by the kernel''s DMA mapping/unmapping API and have > these operations call down into Xen to update the IOMMU > appropriately. On x86 this strategy is likely to be quite costly as > there is little batching to amortize the hypercall to do the mapping, > though unmapping can potentially be done lazily provided the pages > remain pinned. > > Driver domains need to perform IO on behalf of other domains, and the > grant table mechanism gives them the ability to create temporary > mappings to read/write data. To be able to deliver data in a zero copy > fashion, page grants need to be extended to the IOMMU. The grant table > mechanism was designed with this in mind, so it''s a relatively > straightforward extension. When calling into xen to map a grant > handle, a guest can specify whether the page should also be added to > the IOMMU. The grant unmap operation can operate in a similar fashion. > > The only potential issue with this approach is that at the point that > the map operation is performed the device that is going to be doing > the IO may not yet be known (for example for network transmit we need > to do the map operation before we can inspect the packet header and > determine the destination interface). In most situations, it probably > makes sense for all devices under control by the same driver domain to > share the same pagetable structure, so this is not an issue. If this > is not the case, mapping can be deferred to the kernel''s DMA mapping > functions, but there is likely to be less batching to amortize the > hypercall cost. > > > In 32b x86 Xen, address space is at a premium. This leads to the > notion of xen heap pages and separate `domain pages''. The former may > be accessed by Xen rapidly in any context, whereas domain pages must > be mapped dynamically on demand. On x86\_64 this distinction is > redundant as Xen has a 1:1 mapping of all physical memory, and hence > domain pages may be accessed efficiently. However, the current x86\_64 > code inherits the statically sized heap from 32b xen. As an immediate > fix, this heap is arguably too small for a large x86\_64 machine, and > limits the number of VMs that can be started (the limit is lower than > on 32b as the domain structure for VMs is larger on 64b builds). A > better solution might be to unify the xen and domain heaps on 64b > builds. > > > IBM have led the excellent work to add fine-grained access control > mechanisms to Xen''s low-level interfaces. However, the current dom0 > control interface has a very simple `flat'' notion of privilege, and > extending this to allow more flexible delegation of control over > guests would certainly be desirable for some deployment scenarios. > > Today, there already exists a notion of a bitmap of privilege > capabilities that a guest has. (Note that this is orthogonal to the > sets of physical resources it has control over such as ranges of > machine address space, io ports etc). The current capability set is > rather small, and could no doubt be made more fine grained. > > > More interestingly, it would be useful to be able to delegate > privilege such as to be able to grant a domain permission to perform a > certain privileged operation on some specified other domain or group > of domains. This leads naturally to a hierarchical model of > domain resource allocation and permission, for example allowing a > domain with only a very restricted privilege capability to create a > new domain by carving it out of its own resource allocation. It would > then have full control over this domain, allowing it to destroy it, > pause it, map its pages, attach a debugger etc. > > >From Xen''s low-level `datapath'' point of view we want to flatten this > > hierarchy to keep the privilege check operations as simple as > possible, with only the control operations having to worry about the > extra complexity. Citing the example in the previous paragraph of > having one domain build another, this should be quite achievable as > some care is already taken to have the domain builder use standard > unprivileged interfaces. > > > \section{Testing and Debugging} > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > Users require Xen to be a rock-solid stable system component, > achieving greater stability than the OSes which run on it. Generally, > we don''t do too badly on this front. We''re fortunate to have a > relatively small and tight code base, coupled with significant > investment in testing by a number of parties. > > The tip of the -unstable tree is exercised daily by IBM, Intel and > VirtualIron as well as XenSource. The XenSource test reports are > viewable on the web at http://xenbits.xensource.com/xenrt, while other > reports go to the xen-devel list. The continuous testing provides a > good way of monitoring the progress toward stability of new features, > and for flagging regressions. > > In fact, most potential regressions never make it out into the > -unstable tree, as they are picked up by automated testing in the > staging tree, which gives changesets a grilling on three different > machines (32b, PAE, and x86\_64) before pushing the changesets out to > -unstable. Hard-core developers wanting to see checkins as soon as > they happen can do so by subscribing to xen-staging@lists.xensource.com. > > Prior to the 3.0 release we released a special testing CD and > encouraged users to download and run it on their hardware then upload > the results. The CD provided booted a native Linux kernel followed by > 32b, PAE and x86\_64 kernels (as appropriate for the hardware), running > a whole series of tests and benchmarks under each and recording the > results to a log file which was then uploaded. The test CD proved very > useful identifying machines Xen struggled on, and helped us get many > of the issues fixed prior to release. Post release, the 3.0 demo CD > has proved useful in debugging various platform issues reported by > users, enabling them to collect and supply developers with data from > both native and xen kernel boots. > > One avenue we are investigating to help find bugs is submitting Xen to > the OSS code scanning programme run by Coverity with funding from > DARPA. Coverity''s tools have proved useful in finding bugs in a number > of other kernel-level projects, so it makes sense to investigate. The > tool will likely generate a long list of possible issues, which will > require a concerted effort from the community to investigate and > classify, and fix the subset that are real bugs. > > Even with the best possible test and QA programme, some number of > in-field crashes will be sadly inevitable. In these circumstances, > it''s important that developers can gather as much information as > possible from the incident. For user-space incidents, `xen-bugtool'' is > useful for collecting log files and system details. For more serious > failures of domain 0 or xen itself we can''t rely on things getting > logged in standard files, and need support for taking a system core > dump. Horms is doing great work to get kexec working in domain 0, > meaning that we will be able to use kdump to write out a system core. > > Dumping a full copy of system memory can be quite time consuming on a > machine with lots of RAM, so we will possibly have to consider putting > a little more intelligence in the dump routines to harvest the most > useful information e.g. a quick dump that concentrates on CPU register > state and stack information for xen and dom0, and a more comprehensive > dump that collects all xen and dom0 state and just register and stack > info for other guests. We certainly need tools to help pick apart > these dumps and turn them into a form that gdb can load to assist > examination. > > For xen developers there are a number of debugging aids that can > called on. There is a serial gdb stub that can be connected to > remotely and used to examine both xen and domain 0. Since this halts > the system while the debugger is connected, it typically can''t be used > on production systems. One frequently useful tool is the debug > console, which by default is accessed by hitting ctrl-A three times on > the serial console. Hitting `h'' gives a help menu of features the > console can perform. It''s main use is for diagnosing the state of the > system in event of a hang. Register dumps can be used to see what all > the CPUs are doing, whether guests are servicing interrupts etc. The > console also provides access to the software performance counters and > other > statistics. > > Debugging guest VMs can be achieved using Xen''s gdbserver support. > When gdbserver is started in dom0 against a particular VM, it > effectively implements gdb serial stub functionality on behalf of the > guest, enabling gdb to connect to the gdbserver via a TCP > port. gdbserver has recently been extended to support both PV and HVM > guests. If getting register and stack state for all the VCPUs from a > guest is all that''s required, the `xenctx'' command is quicker than > firing up gdb. > > When guest VMs crash we currently have the option to preserve the VM, > enabling debugging via gdb or xenctx. However, it would be useful to > have the option to write out a core image of the guest VM. This is not > quite the same as doing a VM save-to-disk operation, as we would like > to write the image out in a form that gdb could then inspect. The old > xen 1.2 tools had basic guest core dump support, and it would be good > to see this feature back in Xen 3. > > > > \section{Misc} > %%%%%%%%%%%%%% > > We need to add a hypercall to xen to allow a guest to extend the size > of it''s grant table. This should be straightforward, but in its > current absence is the restriction which leads to the current ``max 3 > VIF''s per guest'''' limit. Actually, this restriction could also be > solved by making the allocation of grant table entries to a VIF > dynamic, as already happens with the blkfront driver. We should > implement both strategies forthwith. > > > It would be useful to add PV extensions to guests to assist in taking > consistent filesystem snapshots, for example, when creating ``template > VMs''''. The obvious way of doing this would be to extend the xenbus > mechanism used to deliver shutdown requests and `magic sysrq'' > operations. Adding the ability to issue ``sync disks and pause'''' and > ``remount filesystems read-only and pause'''' would be useful. > > > We need to make a few simple extensions to the control API to allow > the set of CPU feature flags exposed to a guest VM (both HVM and PV) > to be ``cooked'''' such that features may be hidden. This is useful in a > heterogenous CPU environment where it may be desirable to only allow > guests to see a lowest common denominator set of flags such that > save/restore/relocate images remain portable. > > > The linux guest code currently implements a strict priority ordering > over how it services pending event channels. We may wish to replace > this with a scheme that uses round-robin servicing of events within > each group of 32 event channels, thus providing a hybrid approach that > supports both prioritization and fairness. This is purely a guest > issue, and the current strict priority scheme doesn''t seem to cause > any problems today. > > > Currently, each PV guest is allowed just a single virtual serial > console device. Although once we have PV framebuffer support (and > associated virtual terminals) virtual serial console support will be > less important, it would still be nice to support multiple consoles > per guest. This should be relatively straight forward, requiring > updates to the console frontend driver, xenconsoled, and the tools. > > > When using `xm mem-set'' commands to control the amount of memory in a > guest its currently quite easy to set the target too low and create a > `memory crunch'' that causes a linux guest kernel to run the infamous > `oomkiller'' and hence render the system unstable. It would be far > better if the interaction between the balloon driver and linux''s > memory manager was more forgiving, hence causing the balloon driver to > `back off'', or ask for more memory back from xen to alleviate the > pressure (up to the current `mem-max'' limit). The hard part here is > deciding what in the memory management system to trigger off -- at the > point where the oom killer runs the system is typically already > unusable, so we want to be able to get in there earlier. Seeking > advice from Linux mm gurus would be helpful. > > When PV guests boot, the kernel and initial ram disk images must be > supplied to the domain builder. This is not unlike what happens when a > physical machine boots using PXE. However, from a maintenance point of > view it is very convenient to store the kernel and initrd in the guest > virtual disk image, where it can be kept in-sync with its kernel > modules. This has led to a couple of different schemes to read the > kernel and initrd from out of the guest filesystem as part of the PV > guest domain building process. pygrub uses libext2/libreiser to read > the images out of the guest filesystem, whereas domUloader mounts the > guest filesystem in dom0 to extract the files. The latter suffers from > potential security issues in the presence of maliciously crafted > filesystem images, but is otherwise simpler to set up. > > A better solution to this problem would be to use a bootloader run > within the guest domain. This could be done using a cut-down linux > instance that kexec''s the final guest kernel, or by porting a > bootloader such as grub to be able to use the PV net/block > devices. Grub2 has quite wide filesystem support and is considerably > easier to work on than the old grub code, so adding these drivers may > not be that difficult. Grub supports a wide range of filesystems, > including UFS that used by Solaris. [NB: is UFS supported in Grub2?] > > The save/restore/relocate in Xen provides almost all we need to be > able to take copy-on-write snapshots of VM''s memory, to be used for > rollback or to checkpoint long running jobs (if the guest is > communicating with other machines then the wider effect of such a > rollback must be considered). To be able to support checkpoints, we > need to extend the tools to coordinate snapshotting of virtual disks > with taking the execution state snapshot. As well as enabling such > checkpoints to be initiated from the control tools, perhaps providing > the ability to trigger them from within the guest would be useful too. > A natural progression from supporting checkpointing would be to > enable ``VM forking''''. Rather than creating a read-only checkpoint, > the VM effectively becomes cloned, running in a different domain, > writing to a CoW snapshot of the disk. For VMs that have network > access a mechanism is needed for enabling the IP configuration of the > guest to be updated for the cloned VM. It should be possible to add > code in the PV `resume'' path to change the IP address and update > listening sockets while closing open connections. > > Xen currently lacks support for devices that require ISA DMA (DMA > below 16MB). Although ISA devices are rare these days, some PCMCIA > cards have the same restriction (particularly older WLAN cards), > creating problems for some laptops. We need to assess whether adding > such support is going to be worth the effort or not. > > Adding support for call graph support into xen oprofile would be > useful. This would involve capturing the first few items on the stack > as well the EIP (though there would be issues when frame pointers are > missing). > > Hardware performance counters are currently considered to be > system-wide, which works great for xen oprofile, but prevents guests > from profiling themselves. Adding support to allow performance > counters to be efficiently contexted switched between domains would be > useful (though use in this manner would be mutually exclusive with > system-wide use). > > Although efforts are made to ensure the security of the xen hypercall > API, it wouldn''t hurt to do a complete code audit: it is critical that > unprivilged guests should not be able to access data that doesn''t > belong to them, or crash the system. Efforts should be made to bound > the scope of ''denial of quality of service'' attacks too. > > A useful tool to help test the integrity of the xen guest API would be > a xen equivalent of the ''crashme'' tool used for testing the Linux > process API. The tool would attempt to upset Xen by operating like a > normal guest but using random numbers to perturb hypercall arguments, > make ''difficult'' corruptions to pagetables and other structures etc. > (e.g. mutually recusrsive pagetables would be ain interesting one). > > Finally, its about time we did another iteration on the user manual, > wiki and other documentation to ensure its up to date and relevant. > > > \section{The IA64 ports} > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > %[insert info supplied by maintainers] > > The IA64 port is making excellent progress with contributions from HP, > Intel, Fujitsu, Bull and VALinux to name but a few. Base performance > is currently excellent, incurring an typical overhead of around 2\% on > native. Multiple Linux domains are supported (including support for > virtual SMP), and clean shutdown has been completed. In addition the > source code has been reorganized to produce a clean abstraction layer > at the source level; with this patch applied, the resulting kernel > binary can be run both under Xen and on bare metal. > > > Xen/ia64 is also getting close to feature parity with the x86 > ports. Virtual block device using the standard backend/frontend > model has been integrated, and the same set of control tools > and commands are used to manage the system. > > There is also support for VT-i --- the hardware virtualization > e-technology for IPF --- which allows the running of completely > unmodified guest operating systems along-side paravirtualized > domains. Once again, the control tool set has been aligned with > the VT-x model to maximize compatibility. > > In terms of hardware compatibility, it has been extensively > tested to operate on the following configurations: > \begin{itemize} > \item Intel Tiger4 > \item HP rx2600/rx1600/rx2620 (and likely any HP zx1 based system) > \item Bull NovaScale 4000 and 6000 systems, with 5000 to follow > \item Fujitsu Primequest series > \end{itemize} > > In terms of future work, the main areas are: > \begin{enumerate} > \item Completing/stabilizing the work to give each guest a > virtual physical address space; > \item Implementing save/restore and migration; > \item Providing support for driver domains; and > \item Additional stability / performance testing and improvements. > \end{enumerate} > > > \section{The PowerPC Port} > > The PowerPC work, led by IBM, is focusing on the PowerPC 970 processor > which includes hardware extensions designed to support paravirtualized > operating systems. An additional parallel development task is looking > at getting Xen running on 970s with hypervisor mode disabled (e.g.\ > Apple G5 systems). > > The port doesn''t use the standard split driver model yet, but > is running guests. We hope that the code can get > merged into -unstable in the next couple of months; various changes to > support this forthcoming merge have already been incorporated (e.g. > the use of {\tt copy\_from\_guest()} which on PowerPC uses > physically addressed scatter/gather lists). Further changes to common > code will also be required, which may modify the dom0 API, and so > we need to ensure that people have advance warning before any such > changes go in. > > Power Xen currently only runs on the Maple 970 processor evaluation > board, but support for IBM JS21 blades and other 970-based machines > will follow soon. > > > \section{Aims for next two releases} > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > \subsection{3.0.3 aims; end July to synchronize with FC6 freeze} > \begin{itemize} > \item new CPU scheduler > \item cow file-backed virtual hard disk support > \item qemu-dm updated to latest qemu version, stored as patch queue > \item basic NUMA memory allocator support > \item kexec support > \item xend VM life-cycle management > \item new shadow pagetable code? > \end{itemize} > > \subsection{3.0.4 aims; end Sept} > \begin{itemize} > \item new xml-rpc control API > \item simple storage management in xend > \item qemu `v2e'' integration > \item PV network virtualization improvements > \item PPC merge? > \item PV USB support ? > \end{itemize} > > \section{Task priority list} > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% > > > {\em Schedule:} \\ > 1 - 3.0.3: end July \\ > 2 - 3.0.4: end Q3 2006 \\ > 3 - Q4 2006/Q1 2007 \\ > 4 - beyond \\ > > These priority ratings indicate where various features could be > targeted to land on the roadmap. It reflects where the xen core team > is planning on concentrating its effort, coupled with input from work > that we already know is happening in the community. Obviously other > folk will have different ideas of priority and will scratch their own > itch and submit patches. We should obviously try to iterate in order > to get such patches included ASAP, regardless of the below suggested > schedule. > > \small > > \begin{tabular}{|l|l|p{10cm}|} > \hline > > Sched & Area & Description > \\ > > \hline > > 1 & tools & xend VM life-cycle management > \\ > 1/2 & tools & XML config file and conversion tools > \\ > 1/2 & tools & standardized xen control API: xml-rpc over > https/unixdomain sockets \\ > 1/2 & tools & C++/perl/python bindings for control API > \\ > 2 & tools & simple storage management in xend > \\ > 3 & tools & revive guest coredump support > \\ > 3 & tools & split VM relocation operation into two parts > and authenticate \\ > 3 & tools & DMTF CIM providers > \\ > 3/4 & tools & Web GUI for Xen > \\ > > \hline > > 1 & storage & blktap (or other) support for file-based > virtual disk storage. \\ > 2 & storage & blktap plugins for common formats > \\ > 2 & storage & optimized qcow implementation > \\ > 3 & storage & consider adding write accounting/throttling on > current loop driver \\ > 3 & storage & support for block IO QoS. Use CFQ and ionice, > or implement in blkback? \\ > 3/4 & storage & ''ioctl'' support between blkfront/back > \\ > 3/4 & storage & media change, size change event propagation to > guest userspace \\ > 4 & storage & consider SCSI level storage virtualization > option \\ > > \hline > > 1/2 & network & TCP Segmentation Offload support in device > channel \\ > 2 & network & checksum offload cleanup > \\ > > 2 & network & hypervisor chooses to copy vs. page flip > \\ > 2 & network & dynamic allocation of grant table entries; > grant table resize > \\ > 2 & network & investigate whether bridge code needs to be > ''streamlined'' \\ > 2 & network & jumbo frames support in dom0 and device > channel \\ > 3/4 & network & investigate static shared buffer approach > \\ > 3/4 & network & TCP Offload Engine support in device channel > \\ > 3/4 & network & investigate high-performance point-to-point > link support \\ > 4 & network & RDMA support in device channel > \\ > \hline > > 1 & xen & extensive benchmarking and perf tuning > \\ > 1 & xen & CPU scheduler that balances VCPUs, implements > weight \& caps \\ > 1/2 & xen & initial NUMA mechanism checkin > \\ > 2 & xen & live relocation tuning, robustification, tools > safety interlock \\ > 2/3 & xen & support for running 32b PAE guests on a 64b > hypervisor \\ > 2/3 & xen & improved NUMA policy code > \\ > 2/3 & xen & add order$>$0 guest memory allocation > accounting \\ > 2/3 & xen & extend x86\_64 heap size; merge xen and domain > pools \\ > 3 & xen & investigate bad pre-emption > avoidance/mitigation strategies \\ > 3 & xen & add superpage support for PV guests > \\ > 3 & xen & IOMMU support: isolation of devices to > domains; grant table integration \\ > 3/4 & xen & lazy memory allocation for live relocation of > ballooned guests \\ > 4 & xen & fine-grained delegation for dom0ops; > hierarchical resource model \\ > 4 & xen & power management enhancements: CPU sleep, freq > scaling \\ > 4 & xen & power management enhancements: > suspend/hibernate \\ > 4 & xen & accounting and billing time IO domains spend > on behalf of guests \\ > > \hline > > \end{tabular} > > \begin{tabular}{|l|l|p{10cm}|} > > \hline > > Sched & Area & Description > \\ > > \hline > > > 1 & hvm & fix current shadow pagetable code, add > PAE-on-PAE mode, SMP safety \\ > 1 & hvm & upgrade QEMU version, maintain as a patch > queue \\ > 1/2 & hvm & rewrite shadow pagetable code to optimize, > simplify \\ > 1/2 & hvm & finalize interface for making hypercalls from > VT guests \\ > 2 & hvm & HVM save/restore support; qemu, xen, and tools > changes \\ > 2 & hvm & basic SMP HVM guest support; ACPI tables, > locking safety \\ > 2/3 & hvm & change QEMU-xen interface to use the ''v2e'' > approach \\ > 2/3 & hvm & SMP HVM guest performance and scalability > \\ > 2/3 & hvm & support real superpage mappings for HVM guests > (after adding accounting) \\ > 2/3 & hvm & implement high-performance SCSI HBA emulation > \\ > 2/3 & hvm & implement high-performance Ethernet emulation > \\ > 3 & hvm & live relocation. Add log-dirty support > \\ > 3 & hvm & move QEMU into a ''stub domain'' linked against > a linux kernel \\ > 3/4 & hvm & move QEMU into a ''stub domain'' linked against > a minios \\ > 4 & hvm & HVM hotplug CPU emulation > \\ > > \hline > > 1 & linux & extensive benchmarking and perf tuning > \\ > 1/2 & linux & SMP scalability improvements > \\ > 1/2 & linux & work to get xen in to kernel.org linux > \\ > 2 & linux & code review of x86\_64 port > \\ > 2 & linux & investigate proposed x86\_64 optimizations > \\ > 2 & linux & improve interaction between balloon driver and > page allocator to avoid memory crunch\\ > 2/3 & linux & support for multiple virtual serial consoles > \\ > 3 & linux & consider hybrid round-robin/priority scheme to > service event channels \\ > > \hline > > 1/2 & client & basic kernel fbdev paravirtual framebuffer > implementation \\ > 2/3 & client & USB virtualization; investigate USB-over-IP > code \\ > 2/3 & client & Xserver support for ''h/w cursor'', copy rect > and fill rect \\ > 3/4 & client & OpenGL/Direct3D virtualization > \\ > > \hline > > 2/3 & misc & support for dom0 kexec/kdump to get a machine > core \\ > 3 & misc & infiniband direct guest IO support (finalize > interface, merge xen-smartio.hg) \\ > 3 & misc & support for hiding CPU feature flags from > guests (PV and HVM) \\ > 3/4 & misc & smart NIC direct guest IO support > \\ > 3/4 & misc & submit xen for scanning by Coverity tool; > investigate warnings flagged \\ > 3/4 & misc & tools support for doing auto CPU/memory > resource allocation across VMs \\ > 3/4 & misc & support to checkpoint/rollback guests > \\ > 4 & misc & port Grub2 bootloader to net/blockfront > devices \\ > 4 & misc & investigate ''pluggable driver architecture'' > \\ > 4 & misc & xenfs filesystem-level virtualization; shared > buffer cache \\ > 4 & misc & do we need support for ISA/PCMCIA DMA (below > 16MB)? \\ > \hline > \end{tabular} > > \end{document}-- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
kboot is some kind of initrd that it will be loaded by first kernel. the core is first kernel that has kexec built in. and another kexec util in user space can load the final kernel. I''m using LinuxBIOS->first kernel-->initrd-->kexec the xen.gz. the current kexec already support multiboot image that xen is using. So do you mean the kexec 1. from dom0, that is used to restart xen and dom0 2. or kexec in domU that will load this own first kernel then the final kernel in the domU.? YH On 7/13/06, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:> Just floating this for general comment - > > For the bootloader work, we should consider Kboot: > http://kboot.sourceforge.net/ > > Kboot is designed to use a "conventional" bootloader to start a Linux kernel > running "full" bootloader functionality. This is responsible for loading and > kexec-ing the kernel the user actually wants to run. The advantage of this > is that all the high level functionality of grub (interactive menus, > filesystem support, networking) are now provided by a full Linux environment > rather than a large special purpose codebase. > > Some folks at IBM (Michael included) have been working on an update to this > called kboot_sl (system loader): http://merbach.net/kboot_sl/ > > The zSeries guys want this because their z/IPL bootloader doesn''t have any > interactivity - rather like the domain builder. They want to start a > default "kboot" kernel and have that display the user interface, offer > network boot options, etc, and then kexec to the "real" kernel. Their > requirements are similar to ours, so kboot_sl is worth a look. > > I think this is an approach we should look at. > > It gets us: > * Interactive bootloader > * Guest kernel stored in its own filesystem > * Network boot > * Multiple filesystem support (in principle, anything that Linux supports) > * Exotic boot support (boot from http, ftp, ssh - good for developers) > * A common platform with other software > > A port of Grub(2?) might also be nice to have, but I suspect that (given a > working kexec) kboot_sl will be easier to get working and will be more > flexible. > > Cheers, > Mark >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mark Williamson
2006-Jul-14 11:48 UTC
Re: Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
> So do you mean the kexec > 1. from dom0, that is used to restart xen and dom0 > 2. or kexec in domU that will load this own first kernel then the > final kernel in the domU.?The latter - we currently don''t have a means of implementing a guest bootloader that''s fully isolated (runs only in the guest domain, hence secure in the event of compromise). Kboot will also support network boot, and just about any filesystem we want. Since it runs in the domain, it''ll also support consoles properly for interactive use. The default for config files would be to boot the kboot kernel+initrd, which would then load the correct kernel+initrd for that guest, according to a configuration file or a user menu choice. Cheers, Mark> > YH > > On 7/13/06, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote: > > Just floating this for general comment - > > > > For the bootloader work, we should consider Kboot: > > http://kboot.sourceforge.net/ > > > > Kboot is designed to use a "conventional" bootloader to start a Linux > > kernel running "full" bootloader functionality. This is responsible for > > loading and kexec-ing the kernel the user actually wants to run. The > > advantage of this is that all the high level functionality of grub > > (interactive menus, filesystem support, networking) are now provided by a > > full Linux environment rather than a large special purpose codebase. > > > > Some folks at IBM (Michael included) have been working on an update to > > this called kboot_sl (system loader): http://merbach.net/kboot_sl/ > > > > The zSeries guys want this because their z/IPL bootloader doesn''t have > > any interactivity - rather like the domain builder. They want to start a > > default "kboot" kernel and have that display the user interface, offer > > network boot options, etc, and then kexec to the "real" kernel. Their > > requirements are similar to ours, so kboot_sl is worth a look. > > > > I think this is an approach we should look at. > > > > It gets us: > > * Interactive bootloader > > * Guest kernel stored in its own filesystem > > * Network boot > > * Multiple filesystem support (in principle, anything that Linux > > supports) * Exotic boot support (boot from http, ftp, ssh - good for > > developers) * A common platform with other software > > > > A port of Grub(2?) might also be nice to have, but I suspect that (given > > a working kexec) kboot_sl will be easier to get working and will be more > > flexible. > > > > Cheers, > > Mark-- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Rob Bradford
2006-Jul-14 12:10 UTC
Re: Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
On Fri, 2006-07-14 at 12:48 +0100, Mark Williamson wrote:> The default for config files would be to boot the kboot kernel+initrd, which > would then load the correct kernel+initrd for that guest, according to a > configuration file or a user menu choice.Could we use initramfs rather than a separate initrd? If I understand correctly this is just the kind of use case it would work well for. Cheers, Rob -- Rob Bradford <robster@debian.org> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Michael Loehr
2006-Jul-14 12:40 UTC
Re: Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
Hi Rob, we always talk about an ''initrd'' but in fact this is normally an initramfs ... best regards Michael --- Michael Loehr Dept. A177, Bldg. 55131-12 / UG / 6C Linux for zSeries Development Hechtsheimer Str. 2 55131 Mainz, Germany Phone +(49)-6131-84-3396, Fax -6592, Mobile +(49)-171-860-5354 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Michael Loehr
2006-Jul-14 13:26 UTC
Re: Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
So if I understand you right (i am not so firm in Xen terminology yet) you currently have to load your guest kernel + ramdisk from a bootloader that is running in the ''master'' environment. This would mean you have to keep kernel and ramdisk on a filesystem that is readable for the ''master'' environment. With a kexec based bootloader you would always load a standardized kernel and ramdisk first. This mini-linux already runs in the guest environment and acts as a boot loader. Because it is a complete linux you can easily integrate support for all kinds of strange filesystems or network protocols. Depending on a config file or controlled by user interaction this boot loader linux will load the final guest system. With our kboot_sl stuff we provide a prototype for such a type of boot loader. Werner Almesberger did already something similar with his kboot project which we used as our basis. To clarify the difference between these two flavours of kexec based boot loaders a little bit: - kboot is script based, designed to have a small footprint (uses uClibc, busybox, dropbear) and it is expected that kernel and ramdisks are customized for the target system. - kboot_sl is built around a C program that interprets a configuration file and can start various user interfaces. We decided to give up the size restrictions, switched to glibc, integrated additional tools into the ramdisk and will probably replace busybox and dropbear by the ''real thing'' sometime. We try to provide a more generic system loader ramdisk that can be used without much customization to the target system. So if you want to try the latter don''t hesitate to ask me if anything is unclear! best regards Michael --- Michael Loehr Dept. A177, Bldg. 55131-12 / UG / 6C Linux for zSeries Development Hechtsheimer Str. 2 55131 Mainz, Germany Phone +(49)-6131-84-3396, Fax -6592, Mobile +(49)-171-860-5354 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon
2006-Jul-14 14:27 UTC
Re: Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
On Thu, Jul 13, 2006 at 10:12:47PM -0700, yhlu wrote:> On 7/13/06, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote: > >Just floating this for general comment - > > > >For the bootloader work, we should consider Kboot: > >http://kboot.sourceforge.net/It sounds like this is only going to be usable to boot Linux? regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mark Williamson
2006-Jul-14 14:36 UTC
Re: Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
IIRC, Linux kexec can boot other OSes in principle, so we should be able to make this work for anything Linux can read the filesystem of. Cheers, Mark On Friday 14 July 2006 15:27, John Levon wrote:> On Thu, Jul 13, 2006 at 10:12:47PM -0700, yhlu wrote: > > On 7/13/06, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote: > > >Just floating this for general comment - > > > > > >For the bootloader work, we should consider Kboot: > > >http://kboot.sourceforge.net/ > > It sounds like this is only going to be usable to boot Linux? > > regards > john_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon
2006-Jul-14 15:02 UTC
Re: Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
On Fri, Jul 14, 2006 at 03:36:36PM +0100, Mark Williamson wrote:> IIRC, Linux kexec can boot other OSes in principle, so we should be able to > make this work for anything Linux can read the filesystem of.Which is a pretty serious restriction. I''m still not clear on why this is a preferable approach to modifying grub? thanks, john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mark Williamson
2006-Jul-14 15:20 UTC
Re: Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
> > IIRC, Linux kexec can boot other OSes in principle, so we should be able > > to make this work for anything Linux can read the filesystem of. > > Which is a pretty serious restriction.From the grub info page, supported filesystems are: "BSD FFS", "DOS FAT16 and FAT32", "Minix fs", "Linux ext2fs", "ReiserFS", "JFS", "XFS", and "VSTa fs" Of these, Linux supports: Minix, FAT*, Minix, Reiser, JFS, XFS I understand that the Linux UFS driver can support a number of filesystems, including BSD''s FFS. I think there''s some support for Solaris'' UFS in there also. The UFS driver had experimental write support last time I checked, but all that''s really needed is basic read support to scrape the kernel off the disk. There''s also a Summer of Code project to make ZFS available as a FUSE filesystem, which grub can''t support (at the moment...). But until Solaris uses ZFS as a root fs this probably doesn''t matter so much.> I''m still not clear on why this > is a preferable approach to modifying grub?It''s not necessarily a replacement for grub, but I think it''d be good to consider as an interim measure or as an alternative (there are already patches for kexec-under Xen floating around, so all we''d need to do in principle is to compile a kboot_sl ramdisk and run with it). After all, it''s not like there''s a working grub port (that I know of) at the moment, so it''s not like we lose anything ;-) Is there anything I''ve missed in grub that''s really not supported well enough to be done in Linux? Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Rob Bradford wrote:> On Fri, 2006-07-14 at 12:48 +0100, Mark Williamson wrote: > >>The default for config files would be to boot the kboot kernel+initrd, which >>would then load the correct kernel+initrd for that guest, according to a >>configuration file or a user menu choice. > > > Could we use initramfs rather than a separate initrd? If I understand > correctly this is just the kind of use case it would work well for.we''ve had the same discussion on olpc, and I even built initramfs images for olpc. There are a number of bugs in initramfs up to and including 2.6.17, which you will want to make sure are fixed. They all related to building -- things just don''t work correctly, due to broken build scripts. I just fixed said bugs and kept going, and offered my patches up but was told ''those bugs just got fixed'', so YMMV. But xen seems to be at 2.6.13 right now, so I expect they''re still there. Modulo those bugs, it ought to work fine -- it did for me. ron _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Michael Loehr wrote:> Hi Rob, > > we always talk about an ''initrd'' but in fact this is normally an initramfs > ...From my point of view, there are several things. there''s the old initrd there''s the cpio-format initrams there''s the initramfs you can build into a bzimage, by specifying several directories in the .config file, that automagically get bundled up into the bzImage. This last type is broken (build side) until recently. ron _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon wrote:> On Fri, Jul 14, 2006 at 03:36:36PM +0100, Mark Williamson wrote: > > >>IIRC, Linux kexec can boot other OSes in principle, so we should be able to >>make this work for anything Linux can read the filesystem of. > > > Which is a pretty serious restriction. I''m still not clear on why this > is a preferable approach to modifying grub?let''s not forget Plan 9, eh? Would this approach support plan 9? My guess is ''uh, I dunno''. ron _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Which is a pretty serious restriction. I''m still not clear on why this > > is a preferable approach to modifying grub? > > let''s not forget Plan 9, eh? Would this approach support plan 9? > > My guess is ''uh, I dunno''.Any approaches support plan9 apart from the plan9 bootloader / chainloading? If not, I guess there''s an argument for porting the plan9 loader also (or porting FS drivers to Linux, which would be useful anyhow!) The kboot approach could access 9p filesystems, if that helps. Maybe you could make a FAT partition on the virtual disk and stash the kernel there? I presume Plan9 supports FAT as a lingua franca? Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> let''s not forget Plan 9, eh? Would this approach support plan 9?Sure, as long as the xen binary load file format is still supported. We can load out of a FAT partition. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mark Williamson wrote:> The kboot approach could access 9p filesystems, if that helps. Maybe you > could make a FAT partition on the virtual disk and stash the kernel there? I > presume Plan9 supports FAT as a lingua franca?yeah, as much as I dislike it, the boot partition on plan9, for now, is FAT. Yuck. ron _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon
2006-Jul-14 15:46 UTC
Re: Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
On Fri, Jul 14, 2006 at 04:20:59PM +0100, Mark Williamson wrote:> Of these, Linux supports: Minix, FAT*, Minix, Reiser, JFS, XFS I understand > that the Linux UFS driver can support a number of filesystems, including > BSD''s FFS. I think there''s some support for Solaris'' UFS in there also. The > UFS driver had experimental write support last time I checked, but all that''s > really needed is basic read support to scrape the kernel off the disk. > > There''s also a Summer of Code project to make ZFS available as a FUSE > filesystem, which grub can''t support (at the moment...). But until Solaris > uses ZFS as a root fs this probably doesn''t matter so much.It means every OS with a file system that''s not a Linux one will typically have to write it two times more: for grub, and for Linux (along with making Linux actually work well enough). And of course this doubles the support/maintenance matrix as well.> > I''m still not clear on why this > > is a preferable approach to modifying grub? > > It''s not necessarily a replacement for grub, but I think it''d be good to > consider as an interim measure or as an alternative (there are already > patches for kexec-under Xen floating around, so all we''d need to do in > principle is to compile a kboot_sl ramdisk and run with it).As an interim solution it might be OK, sure. regards john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > let''s not forget Plan 9, eh? Would this approach support plan 9? > > Sure, as long as the xen binary load file format is still supported. > We can load out of a FAT partition.AFAIK it ought to be a case of just making sure the kexec tools can load the Xenified plan9 kernel - we could always patch them if necessary, I guess. Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mark Williamson
2006-Jul-14 16:13 UTC
Re: Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
> It means every OS with a file system that''s not a Linux one will > typically have to write it two times more: for grub, and for Linux > (along with making Linux actually work well enough). And of course this > doubles the support/maintenance matrix as well.Which OSes are you thinking of here? Linux currently supports (assuming it works well enough for read access) the boot filesystems of everything that runs on Xen, plus more. It''s a fair point that it increases effort for OS maintainers if they have a grub driver for a filesystem that Linux doesn''t already support - I''m not clear what this could be though. Are you thinking about ZFS, perhaps?> As an interim solution it might be OK, sure.I suspect given the choice between kboot_sl and grub some users might prefer kboot - it is more featureful (apart from the graphical loading, I suspect it might actually be a superset in terms of features now). We can always make which bootloader is used be configurable (even per domain - perhaps we could have an unprivileged interface), anyhow. Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon
2006-Jul-14 16:16 UTC
Re: Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
On Fri, Jul 14, 2006 at 05:13:37PM +0100, Mark Williamson wrote:> > It means every OS with a file system that''s not a Linux one will > > typically have to write it two times more: for grub, and for Linux > > (along with making Linux actually work well enough). And of course this > > doubles the support/maintenance matrix as well. > > Which OSes are you thinking of here? Linux currently supports (assuming it > works well enough for read access) the boot filesystems of everything that > runs on Xen, plus more.Presuming they all work well enough. Stuff like UFS is the sort of corner code that doesn''t get much love, because so few people use it. Making boot rely on it not breaking seems worrying. To some degree, the same goes for FAT.> It''s a fair point that it increases effort for OS maintainers if they have a > grub driver for a filesystem that Linux doesn''t already support - I''m not > clear what this could be though. Are you thinking about ZFS, perhaps?Yes. regards, john _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mark Williamson
2006-Jul-14 16:26 UTC
Re: Xen bootloader (was: Re: [Xen-devel] Xen Roadmap proposal)
> > Which OSes are you thinking of here? Linux currently supports (assuming > > it works well enough for read access) the boot filesystems of everything > > that runs on Xen, plus more. > > Presuming they all work well enough. Stuff like UFS is the sort of > corner code that doesn''t get much love, because so few people use it. > Making boot rely on it not breaking seems worrying. To some degree, the > same goes for FAT.I think FAT''ll be OK - it has to work reasonably well for people with digital cameras, MP3 players, etc. I agree UFS is a bit less likely to be well maintained, but I imagine if read support works it''s relatively unlikely to break... It''s something that probably would merit investigation before anyone tries to mandate a standard Xen bootloader (if, indeed, that is a good idea in the first place).> > It''s a fair point that it increases effort for OS maintainers if they > > have a grub driver for a filesystem that Linux doesn''t already support - > > I''m not clear what this could be though. Are you thinking about ZFS, > > perhaps? > > Yes.Really, with a suitable bootsequence we could probably have a much more guest-based bootloader setup such that users could specify their own bootloader using a "boot block", which could be quite a nice compromise. The ZFS-FUSE stuff looks like it might help with this, since in userspace it''s able reuse the Solaris code without licensing worries. However, it''s still in the pipeline, without a guaranteed completion date (although it *ought* to work by the SoC deadline, there''s not a lot else to do other than cross fingers here). Until somebody produces a Xenified version of Grub it looks like the most straightforward solution for most people is to use pygrub and friends, or try kboot if they''re feeling adventurous. Once a Grub/Xen version is available I guess we can weight up the pros/cons of different approaches then. I assume you''re using Grub 1 - if it were based on Grub2 presumably the ZFS driver would still need porting? Cheers, Mark _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 7/14/06, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:> > > let''s not forget Plan 9, eh? Would this approach support plan 9? > > > > Sure, as long as the xen binary load file format is still supported. > > We can load out of a FAT partition. > > AFAIK it ought to be a case of just making sure the kexec tools can load the > Xenified plan9 kernel - we could always patch them if necessary, I guess.It seems like the right approach would be to mimic the bios rather than put yet another Linux kernel in the trusted base. Supposedly this new bootloader we are talking about would be part of the TCB, and so shold be small, trustworthy, and stable. Relying on the ability to patch things later on just to support new formats is going in the wrong direction IMHO. I''d opt for porting grub. I already have a format-neutral two-stage domU bootloader which boots Linux from the ramdisk as well as unrolls checkpoints, and writing it was not that hard. The only thing that has kept me from submitting it as a patch* it is the issue of hypercall-page initialisation, but my emails about this to the list have gone unanswered lately. Sure it needs block or network drivers, but starting out with something small rather than the inherently bloated and buggy Linux code seems a much better idea. Oh well, at least people seem to have discovered that dom0 domain building is a dead-end :-) Regards, Jacob * The source is here: http://www.distlab.dk/hg/index.cgi/xen-evilman.hg -- in the extras/cstrap directory. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
John Levon wrote:> On Thu, Jul 13, 2006 at 10:12:47PM -0700, yhlu wrote: > >> On 7/13/06, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote: >>> Just floating this for general comment - >>> >>> For the bootloader work, we should consider Kboot: >>> http://kboot.sourceforge.net/ > > It sounds like this is only going to be usable to boot Linux?See http://www.suse.de/~kraxel/xen/kexec.html ... I''m currently at the point that 32bit non-pae kexec works. The page is slightly outdated, xenbus and device handover to the next kernel works meanwhile, i.e. the kexec''ed kernel has working virtual disk and network. I''ll plan to hack up a very simple (i.e. just disk boot, not interactive) domU boot loader next. First because it is a useful application, also it will make turnaround times for kexec testing much smaller. Two birds with one stone ;) The xc_kexec utility re-useses the (rewritten) domain builder code, in theory it should be able to boot non-linux kernels too, I havn''t even tried that yet though. stay tuned, Gerd -- Gerd Hoffmann <kraxel@suse.de> http://www.suse.de/~kraxel/julika-dora.jpeg _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I think to be successful in the market we still need to focus on stability and we need to make usability a no 1 priority. Whilst there is a regression test suite, it''s not clear how much of the code is actually tested by it and there are areas such as concurrency and error inject which it is clear are hardly tested at all. It would be useful to establish some metrics for code coverage by the test suite and also establish some process for enumerating the tests that need to be created to test any new feature dropped into the tree. We need to be able to continually track how much testing work is outstanding and make sure that we don''t let ourselves get behind in developing regression tests. One of the difficulties with creating the test suite is that it requires expert knowledge of the testing requirements of each area. For the development process to be scalable it would be ideal if developers would contribute a list of tests required to fully exercise any new features introduced (i.e. a test plan) and a set of implemented tests as well. This has been happening already to some extent which is good but I think we need to see a bit more of it. If I were developing a hypervisor from scratch, I would make it self-hosting simply because it would make testing easier: it would be possible to bring up a cluster of hypervisor instances on a single physical machine and, for example, simulate the effect of power failure during inter-machine migration. With this kind of environment and random simulated error injection and random management API calls with probability distribution chosen to maximise code coverage it is possible to create a very effective regression stress test which can set a very high threshold for code quality. We have to start from where we are however and now that Xen is reasonably stable in simple, good path operation, it might be time to start some discussion about how to adapt xm-test or build a stress test environment for testing concurrency, inter-machine operations and error inject including the effects of power failure. On the usability front, I think what is most lacking is a community vision for an out-of-the-box Xen solution. Whilst it is good that on the cutting edge the open source nature of Xen means that it is developing in all directions at once, for market acceptance there needs to be a core idea of a Xen product with well-defined stable supported configurations and a supported feature set that is easy to deploy and ''just works''. It may be the intention to leave this productization step to distributions and other commercial interests but it''s such a lot of work and so important for success in the market that any common core effort on this front can only be beneficial to all parties. So, I''d place emphasis on stability backed by trusted metrics and usability as an explicit common goal. My 2p. Harry. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Seemingly Similar Threads
- [PATCH 0/2] reset xenstore watches to fix kexec in Xen PVonHVM guests
- [PATCH] xen/pv-on-hvm kexec: shutdown watches from old kernel
- Xen max supported memory per guest VM?
- [PATCH v2] xen/console: buffer and show origin of guest PV writes
- question about memory auto-translate and paravirtual and no pseudophysical overlay