Well, the mainline kernel just hit 2.6.27-rc1, so it's time for an
update about what's new with Xen. I'm trying to aim this at both the
user and developer audiences, so bear with me if I seem to be waffling
about something irrelevant.
2.6.26 was mostly a bugfix update compared with 2.6.25, with a few small
issues fixed up. Feature-wise, it supports 32-bit domU with the core
devices needed to make it work (netfront, blockfront, console). It also
has xen-pvfb support, which means you can run the standard X server
without needing to set up Xvnc.
I don't know of any bugs in 2.6.26, so I'd recommend you try it out for
all your 32-bit domU needs. It has had fairly wide exposure in Fedora
kernels, so I'd rank its stability as fairly high. If you're migrating
from 2.6.18-xen, then there'll be a few things you need to pay attention
to. http://wiki.xensource.com/xenwiki/XenParavirtOps should help, but
if it doesn't, please either fix it and/or ask!
2.6.27 will be a much more interesting release. It has two major
feature additions: save/restore/migrate (including checkpoint and live
migration), and x86-64 support. In keeping with the overall unification
of i386 and x86-64 code in the kernel, the 32- and 64-bit Xen code is
largely shared, so they have feature parity.
The Xen support seems fairly stable in linux-2.6.git, but the kernel is
still at -rc1, so lots of other things will tend to break. I encourage
you to try it out if you're comfortable with what's still a fairly high
rate of change.
My current patch stack is pretty much empty - everything has been merged
into linux-2.6.git - so it makes a good base for any changes you may have
Now that Xen can directly boot a bzImage format kernel, distros have a
lot of flexibilty in how they package Xen. A single grub.conf entry can
be used to boot either a native kernel (via normal grub), or a
paravirtualized Xen kernel (via pygrub), without modification.
Fedora 9's kernel-xen package has been based on the mainline kernel from
the outset, but it is still packaged as a separate kernel. kernel-xen
has been dropped from rawhide (what will become Fedora 10), and all Xen
support - both 32 and 64 bit - has been rolled into the main kernel
package.
So, what's next?
The obvious big piece of missing functionality is dom0 support. That
will be my focus in this next kernel development window, and I hope
we'll have it merged into 2.6.28. Some roadblock may appear which
prevents this (kernel development is always a bit uncertain), but that's
the current plan.
We're planning on setting up a xen.git on xen.org somewhere. We still
need to work out the precise details, but my expectation is that will
become the place where dom0 work continues, and I also hope that other
Xen developers will start using it as the base for their own Xen work.
Expect to see some more concrete details over the next week or so.
What can I do?
I'm glad you asked. Here's my current TODO list. These are mostly
fairly small-scale projects which just need some attention. I'd love
people to adopt things from this list.
x86-64: SMP broken with CONFIG_PREEMPT
It crashes early after bringing up a second CPU when preempt is
enabled. I think it's failing to set up the CPU topology properly,
and leaving something uninitialized. The desired topology is the
simplest possible - one core per package, no SMT/HT, no multicore,
no shared caches. It should be simple to set up.
irq balancing causes lockups
Using irq balancing causes the kernel to lock up after a while. It
looks like it's losing interrupts. It's probably dropping
interrupts if you migrate an irq beween vcpus while an event is
pending. Shouldn't be too hard to fix. (In the meantime, the
workaround is to make sure that you don't enable in-kernel irq
balancing, and you don't run irqbalanced.)
block device hotplug
Hotplugging devices should work already, but I haven't really tested
it. Need to make sure that both the in-kernel driver stuff works
properly, and that udev events are raised properly, scripts run,
device nodes added - and conversely for unplug. Also, a modular
xen-blockfront.ko should be unloadable.
net device hotplug
Similar to block devices, but with a slight extra complication. If
the driver has outstanding granted pages, then the module can't be
immediately unloaded, because you can't free the pages if dom0 has a
reference to them. My thought is to add a simple kernel thread
which takes ownership of unwanted granted pages: it would
periodically try to ungrant them, and if successful, free the page.
That means that netfront could hand ownership of those pages over to
that thread, and unload immediately.
Performance measurement and tuning
By design, the paravirt-ops-based Xen implementation should have
high performance. It uses batching where-ever possible, late
pin/early unpin, and all the other performance tricks available to a
Xen kernel. However, my emphasis has been on correctness and
features, so I have not extensively benchmarked or performance tuned
the code. There's plenty of scope for measuring both synthetic and
real-world benchmarks (ideally, applications you really care about),
and try to work out how things can be tuned.
One thing that has already come to light is a general regression in
context switch time compared to 2.6.18.8-xen. It's unclear where
it's coming from; a close look at the actual context switch code
itself shows that it should perform the same as 2.6.18-xen (same
number of hypercalls performed, for example).
This would be an excellent opportunity to become familiar with Xen's
tracing and performance measurement tools...
Balloon driver
The current in-kernel balloon driver only supports shrinking and
regrowing a domain up to its original size. There's no support for
growing a domain beyond that.
My plan is to use hotplug memory to add new memory to the system. I
have some prototype code to do this, which works OK, but the hotplug
memory subsystem needs some modifications to really deal with the
kinds of incremental memory increases that we need for ballooning
(it assumes that you're actually plugging in physical DIMMs).
The other area which needs attention is some sanity checking when
deflating a domain, to prevent killing the domain by stealing too
much memory. 2.6.18-xen uses a simple static minimum memory
heuristic based on the original size of the domain. This helps, but
doesn't really prevent over-shrinking a domain which is already
under memory pressure. A better approach might be to register a
shrinker callback, which means that the balloon driver can see how
much memory pressure the system is under by looking getting feedback
from it.
A more advanced project is to modify the kernel VM subsystem to
measure refault distance, which is how long a page is evicted before
being faulted back in again. That measurement can tell you how much
more memory you need to add to a domain in order to get the fault
rate below a given rate.
gdb gives bad info in a 64-bit domain
For some reason, gdb doesn't work properly. If you set a
breakpoint, the program will stop as expected, but the register
state will be wrong. Other users of the ptrace syscall, such as
strace, seem to get good results, so I'm not sure what's going on
here. It might be a simple fix, or symptomatic of a more serious
problem. But it needs investigation first.
My Pet Project
What's missing? What do you depend on? What's needed before you
can use mainline Xen as your sole Xen kernel?
Thanks,
J
On 31/07/2008 01:51, Jeremy Fitzhardinge wrote:> what''s new with Xen.Thanks for the write-up (and the work itself!)> What''s missing? What do you depend on?I realise pv_ops dom0 should come first to save users'' pain in that dept, currently I use pciback/front on 2.6.18 kernels, any idea when this might re-appear either in domU or dom0 pv_ops kernels? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Wed, Jul 30, 2008 at 05:51:37PM -0700, Jeremy Fitzhardinge wrote:> Now that Xen can directly boot a bzImage format kernel, distros have a > lot of flexibilty in how they package Xen. A single grub.conf entry can > be used to boot either a native kernel (via normal grub), or a > paravirtualized Xen kernel (via pygrub), without modification. > > Fedora 9's kernel-xen package has been based on the mainline kernel from > the outset, but it is still packaged as a separate kernel. kernel-xen > has been dropped from rawhide (what will become Fedora 10), and all Xen > support - both 32 and 64 bit - has been rolled into the main kernel > package.An important thing to note is that support in Xen userspace to boot from a bzImage is fairly new - so if you have any existing Xen based products/distros you should check that it has bzImage support if you want to be guarenteed able to boot mainline kernels. We're pushing updates to existing Fedora/RHEL Xen userspace RPMs to enable bzImage support. IIRC the primary changeset you'll need from xen-unstable is this one: changeset: 17332:db943e8d1051 user: Keir Fraser <keir.fraser at citrix.com> date: Tue Apr 01 10:09:33 2008 +0100 files: tools/libxc/Makefile tools/libxc/xc_dom_bzimageloader.c tools/libxc/xc_dom_elfloader.c description: x86: Support loading Linux bzImage v2.08 and up. The latest -mm kernel (2.6.25-rc3-mm1) contains v2.08 of the Linux bzImage format which embeds an ELF file in place of the raw payload allowing it to be extracted and used by the Xen domain builder. It is expected that this functionality will be put forward for 2.6.26. Signed-off-by : Ian Campbell <ijc at hellion.org.uk> Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Wed, Jul 30, 2008 at 05:51:37PM -0700, Jeremy Fitzhardinge wrote:> > So, what''s next? > > The obvious big piece of missing functionality is dom0 support. That > will be my focus in this next kernel development window, and I hope > we''ll have it merged into 2.6.28. Some roadblock may appear which > prevents this (kernel development is always a bit uncertain), but that''s > the current plan. > > We''re planning on setting up a xen.git on xen.org somewhere. We still > need to work out the precise details, but my expectation is that will > become the place where dom0 work continues, and I also hope that other > Xen developers will start using it as the base for their own Xen work. > Expect to see some more concrete details over the next week or so. >Hi! Are you planning to start with redhat''s dom0-pvops work? http://git.et.redhat.com/?p=linux-2.6-dom0-pvops.git;a=summary It hasn''t been updated for a while, but I guess it should be working to some extend.. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen wrote:> Are you planning to start with redhat''s dom0-pvops work? > > http://git.et.redhat.com/?p=linux-2.6-dom0-pvops.git;a=summary > > It hasn''t been updated for a while, but I guess it should be working to some > extend.. >Yes, that''s the plan. Stephen Tweedie has a half-completed rebase to bring it somewhat up to date, and I''m hoping to use that as the base for further work. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2008-Aug-26 08:30 UTC
Re: [Xen-devel] State of Xen in upstream Linux / pv_ops dom0
On Tue, Aug 12, 2008 at 09:24:22AM -0700, Jeremy Fitzhardinge wrote:> Pasi Kärkkäinen wrote: > >Are you planning to start with redhat''s dom0-pvops work? > > > >http://git.et.redhat.com/?p=linux-2.6-dom0-pvops.git;a=summary > > > >It hasn''t been updated for a while, but I guess it should be working to > >some > >extend.. > > > > Yes, that''s the plan. Stephen Tweedie has a half-completed rebase to > bring it somewhat up to date, and I''m hoping to use that as the base for > further work. >Yep. I noticed some dom0 patches here: http://xenbits.xensource.com/paravirt_ops/patches.hg/rev/e11831de288c Is there pv_ops dom0 hg-tree already set up somewhere? -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2008-Aug-26 15:20 UTC
Re: [Xen-devel] State of Xen in upstream Linux / pv_ops dom0
Pasi Kärkkäinen wrote:> Yep. > > I noticed some dom0 patches here: > http://xenbits.xensource.com/paravirt_ops/patches.hg/rev/e11831de288c > > Is there pv_ops dom0 hg-tree already set up somewhere? >That''s it - I''m working in a patch queue. But this work is still very preliminary (haven''t actually booted it as a dom0 kernel, though those patches don''t break domU or native). J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel