Xen users - Feb 2014 - OpenXCI update (almost ready to release an alpha version)

Some of you may remember my post about OpenXCI quite a while ago. I have 
still been working on it, and am almost ready to release an alpha version.

OpenXCI is a lightweight desktop-oriented Xen/Linux distribution similar 
to XenClient (in other words, a dedicated dom0, onto which you install 
full general-purpose domUs). Unlike XenClient, it does not have any 
support for remote provisioning and is intended for people who just want 
to run multiple operating systems on one computer.


Features of the alpha version:
* Full suport for graphics passthrough
 - Primary passthrough is supported for many AMD/ATI and Intel GPUs, and 
secondary passthrough is supported for all passthrough-compatible GPUs). 
NVIDIA primary passthrough may be added later, although secondary 
passthrough should work for any passthrough-compatible GPU

* Switching of VMs with key combinations, implemented through 
lightweight graphics and input servers somewhat like those in XenClient 2.x
 - Only a single VM may have the GPU passed through, like in XenClient 
1.x/2.x
 - Unlike in XenClient, the graphics server does not include any 
GPU-specific drivers and does not do any filtering of commands sent to 
the GPU, meaning hardware support is much broader than XenClient 2.x
 - Instead of filtering of GPU commands, switching of the display while 
a GPU passthrough VM is running is implemented by running a special 
viewer on the passthrough VM (the viewer runs full-screen when a 
non-passthrough VM, and hides itself when the passthrough VM is 
focused). Both a patched VNC viewer and a viewer using a shared memory 
buffer are supported (the shared memory viewer is only available on 
Linux at the moment; a kernel driver would probably be required on Windows)
 - When no passthrough VM is running, the dom0 kernel's framebuffer 
console drivers are used instead. The framebuffer console drivers are 
automatically unloaded when a passthrough VM is started and re-loaded 
when it is shut down

* Simple GUI based on nano-X and FLTK
 - Currently extremely basic; the only things supported at the moment 
are shutting down the entire system and launching utilities (a terminal 
running xentop is started automatically to provide a list of VMs, as 
well as a terminal displaying the system log)
 - Like in XenClient, it can be switched to with key combinations like a 
normal VM, although it runs on dom0 and is not implemented with a 
separate UIVM (unlike XenClient 1.x/2.x)
 - The nano-X server is a client of the graphics and input servers; no 
guest VMs display through it (unlike NxTop and XenClient Enterprise, 
where both the dom0-based GUI and guest VMs display through an X server 
on dom0)

* xl toolstack (I had originally planned to use the XCI toolstack, hence 
the name OpenXCI, but xl is similar enough to the XCI toolstack that 
there wouldn't be much of an advantage to switching)
 - Support for OpenXCI-specific features has been added
 - The libxl PCI device reset code has been replaced with a call to a 
Python script using xend's PCI reset code (the libxl reset code depends 
on the kernel to reset the device properly, and the kernel reset code 
only seems to work on devices supporting FLR; the xend reset code is 
capable of resetting many devices that don't support FLR)

* Currently based on Debian 6.0, Xen 4.1, and Linux kernel 3.9.7 from 
OpenSUSE (this is a kernel based on a forward-ported version of the old 
Xenlinux patches; it seems to be more stable with GPU passthrough than 
the pvops kernels that I tried)
 - Will probably be updated to later versions eventually; I had a 
version based on Xen 4.3, but it hung on me quite often, whereas the Xen 
4.1 version seems to be more stable
 - Support for blktap2 (but not blktap1) was restored (Debian disabled 
it for some reason)


Currently I am just in the process of packaging everything so I can put 
together an ISO. I am going to upload the source to the OpenXCI 
Sourceforge page <http://sf.net/projects/openxci>.

I am running OpenXCI exclusively on my main PC (no multi-boot with 
regular bare-metal OS installs at all), which is based on a Core i7-960, 
Intel DX58SO, and Radeon 5770. I am running Solaris, Linux, Windows XP, 
NetBSD, and FreeBSD domUs (with both an Ubuntu domU and a WinXP one 
configured for GPU passthrough; I can only run one at a time since I 
have only one graphics card). There are still a few stability issues 
with it, although dom0 crashes seem pretty rare. The issues mostly seem 
to relate to dom0 giving up and taking back the GPU, as well as some 
with the Ubuntu domU that I'm running; however, one issue I have not had 
is the performance degradation on domU reboot that some people report 
with AMD/ATI GPUs (neither with Linux, nor with Windows). That might be 
because I am using primary passthrough; I'm not quite sure.

I am also in the process of writing another Xen-based hypervisor called 
RT/XH, which will be based on a very different architecture from a 
standard Xen system. Instead of a Unix dom0, it will use a collection of 
stubdoms (there will still be a dom0, but it will be a stubdom as well, 
handling only supervision of other domains; each driver or service will 
run in its own stubdom). These stubdoms will be based on a combination 
of RTEMS and the NetBSD rump kernel (as well as a dynamic linker so that 
programs running in stubdoms don't have to be statically linked with the 
kernel), rather than mini-OS. Backends on domains running normal 
general-purpose OSes will of course still also be supported. The 
paravirtualized I/O interface will be different than that of standard 
Xen as well (it will be more object-oriented and will require less 
boilerplate in frontend drivers), although traditional Xen PV backends 
will still be available. I am also planning to improve Xen's support for 
real-time scheduling (as the name RT/XH might suggest, I am wanting it 
to have proper realtime support so it can be used in embedded systems 
requiring hard realtime, as well as desktops). I am basically just 
intending OpenXCI to be a stopgap until RT/XH is ready, since RT/XH will 
do everything OpenXCI can and more, while being more reliable than Xen 
with a dom0 based on a monolithic-kernel Unix (it will be designed to 
allow crashed drivers to either be auto-restarted or at least fail 
gracefully). I think current Xen systems don't take full advantage of 
the fact that Xen is a microkernel. It's generally not thought of as 
one, but I would consider it to be a kind of microkernel. I would define 
a microkernel as a kernel that only includes scheduling, IPC, and 
low-level memory management, regardless of whether the environment it 
presents to programs running on top of it is based on threads/messages, 
domains/VCPUs/interrupts, or something else entirely.

If anyone here wants to contribute to either OpenXCI or RT/XH, they are 
definitely welcome.