Xen devel - May 2008 - 2.6.25.* domU i386 baremetal/xen dual boot kernels (Fedora 8)

I''ve been having some fun recently with trying to get a stock upstream
kernel working, compiled with support for both baremetal and PV
booting.  This is with a view to getting a similar configuration into
the Xensource automated test setup.

Since I''ve just got such a kernel to boot a Fedora 8 PV guest I
thought I''d share what I did, as there are a couple of tricks to it.
Thanks are due to the guys at Red Hat who have worked hard to get Xen
domU pvops into mainline Linux.

 * Obtain a stock upstream kernel.  I used 2.6.25.3.
 * Make sure that your uname, if you are on a 64-bit system with a
   64-bit kernel (whether user32-on-kernel64 or fully 64), reports
   i486, or set ARCH, so that you build a 32-bit kernel.
 * Configure it.  For the purposes of testing I used a distro kernel''s
   config adjusted for PV operation.  I didn''t to write down which
   exact distro config file I used and what I changed, but it was
   either Fedora 8 or Debian lenny.  I had to turn on Xen guest
   support (CONFIG_XEN) and set High Memory Support to 64GB
   (CONFIG_HIGHMEM64G, which turns on X86_PAE).  I also turned off kvm
   support because my hypervisor happened to be an early 3.3-unstable
   whose cpuid reporting seems to be broken, and attempts by the
   guest''s kvm to use the CPU''s HVM support can crash the
guest.  The
   resulting config file is attached below.
 * Build it.

 * Obtain a suitable initrd.  As far as I can tell, the Fedora tools
   expect you to build an initrd on the target system, and in Fedora 8
   do not appear to support these recent kernels; dependency problems
   seem to preclude just upgrading the initramfs tools.  That means
   that it is necessary to construct an initramfs by hand (or on
   another system).  A Debian initramfs is generally capable of
   booting Fedora 8, so I modified that:
 * I started with initrd.img-2.6.18-5-xen-686 from my Debian lenny
   dom0, which is for these purposes a fairly vanilla install.
 * I then copied the results of
       INSTALL_MOD_PATH=.... make modules_install
   into the initrd''s /lib/modules alongside the existing modules.
 * It was necessary to remove many drivers to reduce the size.  The
   Linux startup code silently ignores an initrd if it is `too large'';
   I don''t know exactly what the threshold is.
       rm -rf sound drivers/media drivers/bluetooth drivers/infiniband \
              drivers/isdn drivers/scsi drivers/usb drivers/video \
              drivers/parport drivers/net drivers/char/drm drivers/ata \
              drivers/hwmon drivers/mtd drivers/md \
   These were just a collection of big drivers I was confident weren''t
   required for booting.  For normal production use, self-hosting, one
   would obviously use the distro''s own initramfs creation tools which
   don''t include huge piles of irrelevant modules.
 * Build the new initrd (not forgetting the  -H newc  option to cpio).

 * Edit the guest filesystem (eg, by mounting it from the host) to:
     - run /sbin/mingetty --noclear hvc0 from inittab
     - have /dev/xvd rather than /dev/sda or whatever in fstab,
       if your guest uses device names rather than labels or uuids.
   The latter means that to make the same guest bootable both
   baremetal and PV means mounting by label or uuid.
 * Create a Xen config file.  You can probably use pygrub to extract
   the kernel from the guest, if you''ve put it there, but I found it
   wasn''t necessary to copy the kernel or its modules into the guest;
   I just loaded it from the host filesystem.  The file to use for
   kernel is the `vmlinux'' from the Linux tree''s toplevel, as
sadly
   the Xen''s loader does not yet support unpicking bzImages.
   You will need
	extra=''console=hvc''
	root=''/dev/xvda6 ro''
   or the equivalent.  Since the kernel and especially the initramfs
   are very large you are likely to have to increase the `memory''
   parameter too.  I have so far only with `vcpus=1''.

Some pitfalls and warnings:

 * Don''t attempt to bypass the need for an initrd by compiling a
   kernel statically, if you are not prepared to go through and remove
   all of the irrelevant drivers.  There are still several drivers in
   the stock kernel which crash when run under Xen (for example
   because they try to do ioremapping without checking to see whether
   their hardware is present).  This can be avoided by building a
   modular kernel, as those drivers will simply not be loaded.

 * As I haven''t yet copied the new kernels'' modules into the
guest,
   the guest''s module loading system hasn''t had the
opportunity to
   load exciting modules which cause things to break.

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, 2008-05-20 at 12:13 +0100, Ian Jackson wrote:
> I''ve been having some fun recently with trying to get a stock
upstream
> kernel working, compiled with support for both baremetal and PV
> booting.  This is with a view to getting a similar configuration into
> the Xensource automated test setup.
FYI within the xen-unstable.hg tree you can build such a kernel with
something like:
        make XEN_TARGET_ARCH=x86_32 XEN_LINUX_NONINTERACTIVE_CONFIG=y
KERNELS="linux-2.6 linux-2.6-git" kernels -j12

The config used will be i386 defconfig + Xen options. I guess that
won''t
necessarily work on an especially wide variety of baremetal platforms
thoguh.

KERNELS can include anything found in xen-unstable.hg/buildconfigs/mk.*

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel