Paul Wouters
2006-Nov-10 04:08 UTC
[Fedora-xen] Booting rootfs instead of disk image, question on xen modules
hi,
I am trying to boot using a rootfs image, instead of a disk image. Just because
it
is much easier to mount filesystems, then to go calculate offsets for mounting.
Also,
because I have various xen2 filesystems I want to use without building virtual
disk
images. On FC5, using kernel= with a *xenU image works fine. On FC6, using the
*xen
kernel image, this does not work. (using known working fc5 xen kernels on fc6
also
didn''t work, but I expected that, due to differences in the xen
hypervisor)
Instead of using bootloader="/usr/bin/pygrub" I manaully set the
kernel, ramdisk
and extra flags, eg:
disk = [
''file:/xen/qtopia.xelerance.net/qtopia.ext3.img,xvda1,w'' ]
kernel="/boot/vmlinuz-2.6.18-1.2798.fc6xen"
root="/dev/xvda1 ro"
extra="ro selinux=0 3 root=/dev/xvda1
initrd=/boot/initrd-2.6.18-1.2798.fc6xen.img"
ramdisk="/boot/initrd-2.6.18-1.2798.fc6xen.img"
(also tried using hda1 instead of xvda1)
But when I boot, I see the following:
[...]
XENBUS: Device with no driver: device/vbd/51713
XENBUS: Device with no driver: device/vif/0
Freeing unused kernel memory: 180k freed
Write protecting the kernel read-only data: 356k
Red Hat nash version 5.1.19 starting
Mounting proc filesystem
Mounting sysfs filesystem
Creating /dev
Creating initial device nodes
Setting up hotplug.
Creating block device nodes.
Loading uhci-hcd.ko module
USB Universal Host Controller Interface driver v3.0
Loading ohci-hcd.ko module
Loading ehci-hcd.ko module
Loading jbd.ko module
Loading ext3.ko module
Loading scsi_mod.ko module
SCSI subsystem initialized
Loading sd_mod.ko module
Loading libata.ko module
Loading ahci.ko module
Loading ata_piix.ko module
Creating root device.
Mounting root filesystem.
mount: could not find filesystem ''/dev/root''
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
switchroot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!
It seems the xenblck and xennet drivers are not being loaded? And indeed,
dissecting the initrd file, I don''t see them. However, grepping the
config
shows they are now inline:
# grep XENB /boot/config-2.6.18-1.2798.fc6xen
CONFIG_XEN_XENBUS_DEV=y
So I am not sure why xen is not picking up the settings - though pygrub
can find them.
The same machine boots fine with pygrub, so it must be something that
I''m doing wrong myself. but even running xm create -n configname did
not give me a clue on what it is that I''m doing wrong.
I''ve attached the output of xm create -n below.
Using config file "/etc/xen/qtopia.xelerance.net".
(vm
(name qtopia.xelerance.net)
(memory 256)
(on_reboot restart)
(on_crash restart)
(vcpus 1)
(uuid 10000007-f66d-374b-29fc-364cd7761111)
(image
(linux
(kernel /boot/vmlinuz-2.6.18-1.2798.fc6xen)
(ramdisk /boot/initrd-2.6.18-1.2798.fc6xen.img)
(root ''/dev/xvda1 ro'')
(args
''ro selinux=0 3 root=/dev/xvda1
initrd=/boot/initrd-2.6.18-1.2798.fc6xen.img''
)
(vncunused 1)
(xauthority /root/.Xauthority)
)
)
(device
(vbd
(uname file:/xen/qtopia.xelerance.net/qtopia.ext3.img)
(dev xvda1)
(mode w)
)
)
(device (vif (bridge xenbr0) (mac 00:16:3e:7a:0e:57)))
)
Paul
Paul Wouters
2006-Nov-10 05:56 UTC
[Fedora-xen] Problem found with workaround, was Re: Booting rootfs instead of disk image, question on xen modules
On Fri, 10 Nov 2006, Paul Wouters wrote: [ ExecSum since it''s a long post: xenU created from xenguest-isntall.py has different initrd from dom0 though with the same version/build/rpmsig, and rpm has a nasty bug] Following up on myself, since i now know what happened:> Loading ohci-hcd.ko module > Loading ehci-hcd.ko module > Loading jbd.ko module > Loading ext3.ko module > Loading scsi_mod.ko module > SCSI subsystem initializedNote the missing "Loading xenblk" which was visible in the boot of the guest that was started with pygrub using a disk image created on the dom0 with xenguest-install.py. I decided to find out where it got loaded from. It couldn''t be from the filesystem, so it had to be initrd. But both were using the same initrd? Well sort of. pygrub extracted one from the diskimage, and I grabbed it from /boot on the dom0. What I found were two different initrd files with the SAME build number. I copied it from xenU to dom0: [root@echelon boot]# ls -l initrd-2.6.18-1.2798.fc6*xen* -rw------- 1 root root 2177639 Nov 10 00:00 initrd-2.6.18-1.2798.fc6xen.img-from-xenguest-install -rw------- 1 root root 1522165 Oct 28 00:34 initrd-2.6.18-1.2798.fc6xen.img-from-dom0 The original date of the xenguest=install version before i copied it was: "Oct 28 16:37" [root@echelon boot]# md5sum initrd-2.6.18-1.2798.fc6*xen* 8b06aa9e364bd1ac5379e652095a4d3e initrd-2.6.18-1.2798.fc6xen.img-from-xenguest-install 6b591a095b5d4e55641d30c25702e272 initrd-2.6.18-1.2798.fc6xen.img-from-dom0 They both claim to come from the package kernel-xen-2.6.18-1.2798.fc6 Both the diskimage based xen''s rpm -V, and the rpm -V on the dom0 claimed the package was not altered? Checking the rpms, I find both as: # rpm -qi kernel-xen-2.6.18-1.2798.fc6 Name : kernel-xen Relocations: (not relocatable) Version : 2.6.18 Vendor: Red Hat, Inc. Release : 1.2798.fc6 Build Date: Mon 16 Oct 2006 03:57:26 PM EDT Install Date: Sat 28 Oct 2006 04:37:14 PM EDT Build Host: hs20-bc2-4.build.redhat.com Group : System Environment/Kernel Source RPM: kernel-2.6.18-1.2798.fc6.src.rpm Size : 46862330 License: GPLv2 So how can they both verify to the same version, with rpm -V not complaining, while containing different initrd images, yet having the same build date? Just to show what I did, so that I won''t go insane: [root@echelon boot]# rpm -qf /boot/initrd-2.6.18-1.2798.fc6xen.img kernel-xen-2.6.18-1.2798.fc6 [root@echelon boot]# rpm -qa|grep kernel-xen kernel-xen-devel-2.6.18-1.2798.fc6 kernel-xen-devel-2.6.18-1.2741.fc6 kernel-xen-2.6.18-1.2741.fc6 kernel-xen-2.6.18-1.2798.fc6 [root@echelon boot]# ls -l /boot/initrd-2.6.18-1.2798.fc6* -rw------- 1 root root 1521809 Oct 28 00:36 /boot/initrd-2.6.18-1.2798.fc6.img -rw------- 1 root root 2177639 Nov 10 00:30 /boot/initrd-2.6.18-1.2798.fc6xen.img -rw------- 1 root root 1522165 Oct 28 00:34 /boot/initrd-2.6.18-1.2798.fc6xen.img-broken -rw------- 1 root root 2177639 Nov 10 00:28 /boot/initrd-2.6.18-1.2798.fc6xen.img-from-xenguest [root@echelon boot]# rpm -V kernel-xen-2.6.18-1.2798.fc6 [root@echelon boot]# cp /boot/initrd-2.6.18-1.2798.fc6xen.img-broken /boot/initrd-2.6.18-1.2798.fc6xen.img [root@echelon boot]# ls -l /boot/initrd-2.6.18-1.2798.fc6* -rw------- 1 root root 1521809 Oct 28 00:36 /boot/initrd-2.6.18-1.2798.fc6.img -rw------- 1 root root 1522165 Nov 10 00:35 /boot/initrd-2.6.18-1.2798.fc6xen.img -rw------- 1 root root 1522165 Oct 28 00:34 /boot/initrd-2.6.18-1.2798.fc6xen.img-broken -rw------- 1 root root 2177639 Nov 10 00:28 /boot/initrd-2.6.18-1.2798.fc6xen.img-from-xenguest [root@echelon boot]# rpm -V kernel-xen-2.6.18-1.2798.fc6 [root@echelon boot]# So either rpm is broken in -qf mode, or it is broken in -V mode... [root@echelon boot]# rpm -qf /boot/initrd-2.6.18-1.2798.fc6xen.img kernel-xen-2.6.18-1.2798.fc6 [root@echelon boot]# rpm -ql kernel-xen-2.6.18-1.2798.fc6 |grep initrd [root@echelon boot]# And as a test: [root@echelon boot]# rpm -qf /boot/initrd-2.6.18-1.2798.fc6xen.img-broken file /boot/initrd-2.6.18-1.2798.fc6xen.img-broken is not owned by any package I guess it is rpm -qf that is broken? Or some advanced "should never be visible" magic is in use. Replacing the dom0 version with the xenguest installed version fixed my problem. The rootfs based xen now boots fine. And this leads me to a final question. Why are the xen modules loaded through initrd to begin with? kernel size can''t be an issue, since all hardware drivers are gone from the xenU - only the xen virtual drivers are needed. Can''t we just build that stuff inline, and save ourselves a lot of trouble booting. It might not be the elegant CS theoretical perfect solution, but it works in many more practical situations, like this one with a voodoo initrd-2.6.18.-1.2798.fc6xen.img appearing. Paul
Chris Lalancette
2006-Nov-10 13:50 UTC
Re: [Fedora-xen] Problem found with workaround, was Re: Booting rootfs instead of disk image, question on xen modules
Paul Wouters wrote:> On Fri, 10 Nov 2006, Paul Wouters wrote: > > [ ExecSum since it''s a long post: xenU created from xenguest-isntall.py > has different initrd from dom0 though with the same version/build/rpmsig, > and rpm has a nasty bug] > > Following up on myself, since i now know what happened: > >> Loading ohci-hcd.ko module >> Loading ehci-hcd.ko module >> Loading jbd.ko module >> Loading ext3.ko module >> Loading scsi_mod.ko module >> SCSI subsystem initialized > > Note the missing "Loading xenblk" which was visible in the boot of the guest > that was started with pygrub using a disk image created on the dom0 with > xenguest-install.py. > > I decided to find out where it got loaded from. It couldn''t be from > the filesystem, so it had to be initrd. But both were using the same > initrd? Well sort of. pygrub extracted one from the diskimage, and > I grabbed it from /boot on the dom0. What I found were two different > initrd files with the SAME build number. I copied it from xenU to dom0: > > [root@echelon boot]# ls -l initrd-2.6.18-1.2798.fc6*xen* > -rw------- 1 root root 2177639 Nov 10 00:00 initrd-2.6.18-1.2798.fc6xen.img-from-xenguest-install > -rw------- 1 root root 1522165 Oct 28 00:34 initrd-2.6.18-1.2798.fc6xen.img-from-dom0 > > The original date of the xenguest=install version before i copied it was: "Oct 28 16:37" > > [root@echelon boot]# md5sum initrd-2.6.18-1.2798.fc6*xen* > 8b06aa9e364bd1ac5379e652095a4d3e initrd-2.6.18-1.2798.fc6xen.img-from-xenguest-install > 6b591a095b5d4e55641d30c25702e272 initrd-2.6.18-1.2798.fc6xen.img-from-dom0 > > They both claim to come from the package kernel-xen-2.6.18-1.2798.fc6 > > Both the diskimage based xen''s rpm -V, and the rpm -V on the dom0 > claimed the package was not altered? > > Checking the rpms, I find both as: > > # rpm -qi kernel-xen-2.6.18-1.2798.fc6 > Name : kernel-xen Relocations: (not relocatable) > Version : 2.6.18 Vendor: Red Hat, Inc. > Release : 1.2798.fc6 Build Date: Mon 16 Oct 2006 03:57:26 PM EDT > Install Date: Sat 28 Oct 2006 04:37:14 PM EDT Build Host: hs20-bc2-4.build.redhat.com > Group : System Environment/Kernel Source RPM: kernel-2.6.18-1.2798.fc6.src.rpm > Size : 46862330 License: GPLv2 > > So how can they both verify to the same version, with rpm -V not complaining, > while containing different initrd images, yet having the same build date? > > Just to show what I did, so that I won''t go insane: > > [root@echelon boot]# rpm -qf /boot/initrd-2.6.18-1.2798.fc6xen.img > kernel-xen-2.6.18-1.2798.fc6 > [root@echelon boot]# rpm -qa|grep kernel-xen > kernel-xen-devel-2.6.18-1.2798.fc6 > kernel-xen-devel-2.6.18-1.2741.fc6 > kernel-xen-2.6.18-1.2741.fc6 > kernel-xen-2.6.18-1.2798.fc6 > [root@echelon boot]# ls -l /boot/initrd-2.6.18-1.2798.fc6* > -rw------- 1 root root 1521809 Oct 28 00:36 /boot/initrd-2.6.18-1.2798.fc6.img > -rw------- 1 root root 2177639 Nov 10 00:30 /boot/initrd-2.6.18-1.2798.fc6xen.img > -rw------- 1 root root 1522165 Oct 28 00:34 /boot/initrd-2.6.18-1.2798.fc6xen.img-broken > -rw------- 1 root root 2177639 Nov 10 00:28 /boot/initrd-2.6.18-1.2798.fc6xen.img-from-xenguest > [root@echelon boot]# rpm -V kernel-xen-2.6.18-1.2798.fc6 > [root@echelon boot]# cp /boot/initrd-2.6.18-1.2798.fc6xen.img-broken /boot/initrd-2.6.18-1.2798.fc6xen.img > [root@echelon boot]# ls -l /boot/initrd-2.6.18-1.2798.fc6* > -rw------- 1 root root 1521809 Oct 28 00:36 /boot/initrd-2.6.18-1.2798.fc6.img > -rw------- 1 root root 1522165 Nov 10 00:35 /boot/initrd-2.6.18-1.2798.fc6xen.img > -rw------- 1 root root 1522165 Oct 28 00:34 /boot/initrd-2.6.18-1.2798.fc6xen.img-broken > -rw------- 1 root root 2177639 Nov 10 00:28 /boot/initrd-2.6.18-1.2798.fc6xen.img-from-xenguest > [root@echelon boot]# rpm -V kernel-xen-2.6.18-1.2798.fc6 > [root@echelon boot]# > > So either rpm is broken in -qf mode, or it is broken in -V mode... > > [root@echelon boot]# rpm -qf /boot/initrd-2.6.18-1.2798.fc6xen.img > kernel-xen-2.6.18-1.2798.fc6 > [root@echelon boot]# rpm -ql kernel-xen-2.6.18-1.2798.fc6 |grep initrd > [root@echelon boot]# > > And as a test: > [root@echelon boot]# rpm -qf /boot/initrd-2.6.18-1.2798.fc6xen.img-broken > file /boot/initrd-2.6.18-1.2798.fc6xen.img-broken is not owned by any package > > I guess it is rpm -qf that is broken? Or some advanced "should never be visible" > magic is in use. > > Replacing the dom0 version with the xenguest installed version fixed my > problem. The rootfs based xen now boots fine. > > And this leads me to a final question. Why are the xen modules loaded > through initrd to begin with? kernel size can''t be an issue, since all > hardware drivers are gone from the xenU - only the xen virtual drivers > are needed. Can''t we just build that stuff inline, and save ourselves > a lot of trouble booting. It might not be the elegant CS theoretical > perfect solution, but it works in many more practical situations, like > this one with a voodoo initrd-2.6.18.-1.2798.fc6xen.img appearing. > > Paul > > -- > Fedora-xen mailing list > Fedora-xen@redhat.com > https://www.redhat.com/mailman/listinfo/fedora-xenPaul, The reason that you are seeing two different initrd''s is that the initrd is actually packaged as a file in the RPM. Since everyone''s hardware is different, including every single driver in an initrd would be a little silly. After the kernel is finished installing, it runs "mkinitrd", looking at your hardware (specifically, /etc/modprobe.conf) and generating an initrd for your particular hardware. In this case, because mkinitrd was run on the dom0, it crucially included the modules needed to run dom0, but not the ones necessary for domU (like, as you said, xenblk and xennet). As far as why they are modules to begin with; I''m not 100% sure, but the policy tends to be to make everything as modular as possible. This cuts down on the size of the kernel, which is, in fact, a concern. Anyway, I hope that helps, and I''m glad you got it working. Chris Lalancette