Fedora xen - Dec 2006 - Getting Fedora yum-installed domU running on fedora with fedora-xen packages

Hi

I wrote a very similar question some hours ago for Debian, which maybe
got no replies becuase nobody is interested in running Debian on
fedora.

The sad thing is, with the same problems and effects, I also don''t get
a Fedory system running that I instelled with yum.  So, just replace
debian with fedora 6 and debootstrap with yum in the mail below.

Again, This same system runs fine with Xen from xensource, but I''d
like to use the Fedora Xen packages, for the security updates and
stuff.

There must be something special in the Fedory Kernel that does make
systems from "normal" partitions not work somehow, or something like
that.


---------- Forwarded message ----------
From: Henning Sprang <henning_sprang@gmx.de>
Date: Dec 18, 2006 1:27 AM
Subject: Getting Debian domU running on fedora with fedora-native
packages (works with xensource binaries)
To: fedora-xen@redhat.com


Hi,
I built a Debian domU on Fedora using debootstrap (which is really
easy, when using alien to create a debootstrap rpm).

This VM runs very fine on fedora fc 6 with xen and the xen-linux
kernel provides by xensource, and when built from sources.

When trying to use it with the fedora xen packages and kernels, I have
absolutely no luck.

First, I was hanging  on error message like "could not mount root
device on /dev/root" (sorry, I don''t have it at hand, becaue I
just
removed the fedora packages and used the xensource packages again, to
be sure it really still works there).

I got past that by creating an initrd  with --with=xennet
--preload=xenblk options, and boot the domU with this instead ofthe
initrd that comes with the fedora xen kernel package.

But then, the domU hangs on "starting crond..."  I have no idea what
is missing there, I already installed udev additionally, but that
doesn''t help.

As said, this same system runs with no problem on FC6 with the
xensource binaries.

BTW: I run the system without pygrub and not with the kernel from
disk, but with the kernel= options...

Henning

On Mon, Dec 18, 2006 at 04:36:18PM +0100, Henning Sprang
wrote:
> Hi
> 
> I wrote a very similar question some hours ago for Debian, which maybe
> got no replies becuase nobody is interested in running Debian on
> fedora.
> 
> The sad thing is, with the same problems and effects, I also don''t
get
> a Fedory system running that I instelled with yum.  So, just replace
> debian with fedora 6 and debootstrap with yum in the mail below.
> 
> Again, This same system runs fine with Xen from xensource, but I''d
> like to use the Fedora Xen packages, for the security updates and
> stuff.
> 
> There must be something special in the Fedory Kernel that does make
> systems from "normal" partitions not work somehow, or something
like
> that.
Most likely not got the correct set of modules built into the initrd.
Upstream Xensource configs have most things built statically into the
monolithic kernel. Fedora practice is to build everything as a module
whenever possible, so we do this on Xen kernels, as well as baremetal.
Or perhaps the Debian kernel utils aren''t new enough to work on a
2.6.18 kernel ?

Finally, are there any SELinux AVC error messages in Dom0 logs ?

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

Hi - thanks for the reply!

On 12/18/06, Daniel P. Berrange <berrange@redhat.com>
wrote:
> Most likely not got the correct set of modules built into the initrd.
> Upstream Xensource configs have most things built statically into the
> monolithic kernel.
Upstream Xen uses a lot of modules, but I rebuilt the sources without
using modules. As that already worked, I did not yet try with the
default xensource binaries - I gonna try that later and see if it also
works, or if I just by accident built the only kernel allowing me to
run debian on fedora.
>Fedora practice is to build everything as a module
> whenever possible, so we do this on Xen kernels, as well as baremetal.
> Or perhaps the Debian kernel utils aren''t new enough to work on a
> 2.6.18 kernel ?
Maybe, but sounds unlikely - when running on Debian, I use the debian
Xen kernel version 2.6.17. but they also provide a 2.6.18 series
kernel for non-xen and xen. So the tools for these kernek Versions
must be there.
> Finally, are there any SELinux AVC error messages in Dom0 logs ?
I don''t see many selinux messages, I admit I have disabled selinux
(I''m used to this from older fedora versions).

But what I see now is that the dom0 messages log is full of "4gb seg
fixup, process ..." messages. On xen-users there is a hint on setting
hwcap 0 segneg, but this is already done by a file in /etc/ld.so.conf,
provided by  the Fedora xen packages I guess.

I also tried further with a Fedora guest installed with yum, and this
works in the meantime. Seems I probably did something wrong before,
too hectic, because now it looks like making an own initrd with
--with=xennet and --preload=xenblk seemed to help here.
When this is running, also in the domU the logs and console are filled
with the 4g seg fixup messages.

Still, with including these modules, Debian is hanging at boot after
printing "Starting periodic command scheduler: crond."

Nothing happening anymore after this point.

Henning

On Mon, 18 Dec 2006 16:36:18 +0100
"Henning Sprang" <henning_sprang@gmx.de> wrote:
> There must be something special in the Fedory Kernel that does make
> systems from "normal" partitions not work somehow, or something
like
> that.
Not on my system at work. We installed a 32 bit hvm FC6 on a physical
disk via virt-manager, and it worked fine under x86_64 dom0 (also
FC6). I did manage to screw it up somehow trying to get the kernel
updated, but it was working great before that :-).

Hi Tom,

On 12/18/06, Tom Horsley <tomhorsley@adelphia.net>
wrote:
> [...]
> Not on my system at work. We installed a 32 bit hvm FC6 on a physical
> disk via virt-manager, and it worked fine under x86_64 dom0 (also
> FC6). I did manage to screw it up somehow trying to get the kernel
> updated, but it was working great before that :-).
Seems I did not explain correctly what I meant.

With what you do, HVM Installs take place exactly not into partitions
you defined from the outside, but into a block devices, and the hvm
system then does full partitioning inside this.

What I mean what fedora kernels have problems with is when pointing
the disk config directly onto a partition created from outside the
domU.

An example:

In your setup you probably have a disk config line like this;

disk=[ ''phy:/dev/hdb1,hda,w'']

What I mean is a setup that look like this:

disk = [ ''phy:/dev/hdb1,hda1,w'']

where I create the ext3 filesystem from the dom0. These are mostly
used with paravirtual systems, because on hvm you just use isos. But
as I already said, these things can be handled by making a special
initrd... Still, a Debian guest won''t work, but works with no problem
with a Kernel from xensource.

Henning

On Tue, 19 Dec 2006, Henning Sprang wrote:
> as I already said, these things can be handled by making a special
> initrd... Still, a Debian guest won''t work, but works with no
problem
> with a Kernel from xensource.
There are many things that are unfortunately biting you when mixing
debian and fedora kernels/xen. Unfortunate, because this is exactly
what endusers of xen hope to archive.

I found the entire debian mkinitrd quite messy. they seem to no longer
have mkinitrd, only as part of the yaird package, which is one that
can only really generate an initrd image for the running kernel (yeah
right, then i likely don''t need it anymore eh?).

Using an initrd created with fedora on debian fails because they seem to
be handling the change of root from initrd different. Might be related
to pivot root changes?

On top of that, we ran into the pae vs no pae issue as well between
these systems, so that we also could not boot fedora kernels/initrds
from a debian xen0 either.

All in all, our migration from xen2 to xen3 has caused us many headaches,
and so far, the xen3 setup is less flexable then we had before. I think
if xen2 had been available for kernels > 2.6.12, I would have backed out
halfway and gone back to xen2.

I did however, get some more appreciation for the additions that redhat
has added to the xensource default builds. Perhaps I should phase out
our < 512MB debian xen0''s for some stripped down version of FC6 that
fits
in the same amount of space.

Paul

On Tue, Dec 19, 2006 at 12:06:42PM +0100, Henning Sprang
wrote:
> Hi Tom,
> 
> On 12/18/06, Tom Horsley <tomhorsley@adelphia.net> wrote:
> >[...]
> >Not on my system at work. We installed a 32 bit hvm FC6 on a physical
> >disk via virt-manager, and it worked fine under x86_64 dom0 (also
> >FC6). I did manage to screw it up somehow trying to get the kernel
> >updated, but it was working great before that :-).
> 
> Seems I did not explain correctly what I meant.
> 
> With what you do, HVM Installs take place exactly not into partitions
> you defined from the outside, but into a block devices, and the hvm
> system then does full partitioning inside this.
Paravirt installs should work in the same way - you are mapping through
virtual disks, not virtual partitions. The latter just happens to work
by chance, and is not supported.
> What I mean what fedora kernels have problems with is when pointing
> the disk config directly onto a partition created from outside the
> domU.
Try mapping your disk through as ''xvda'' instead of
''hda1''.
> An example:
> 
> In your setup you probably have a disk config line like this;
> 
> disk=[ ''phy:/dev/hdb1,hda,w'']
> 
> What I mean is a setup that look like this:
> 
> disk = [ ''phy:/dev/hdb1,hda1,w'']
Two issues there

1. We strongly recommend *against* specifying  hdX / sdX for virtual disk 
   device type in paravirt guests. The Xen XVD driver is hijacking the 
   IDE / SCSI drivers to do this hack. This will not be accepted upstream
   and so support will likely disappear in a future kernel-xen release.
   Use xvdX  instead, eg

     ''phy:/dev/hdb1,xvda,w''

2. The first form of disk config you have is the preferred setup. ie
   map the partition from Dom0, through to DomU as a *disk*, not a
   partition. DomU should then partition this further as required,
   Your second config ''phy:/dev/hdb1,xvda1,w'' may /happen/ to
work, but
   there are no guarentees this will continue to work in the future.
   Only the ''phy:/dev/hdb1,xvda,w'' format is supported for
compatability
   long term.

    eg, /dev/hdb1 in Dom0, maps to /dev/xvda in DomU
        then DomU  can create /dev/xvda1 as needed

   By having nested partition tables in this way, you ensure that Dom0
   does not see the individual partitions from the DomU. If it were
   to see the indiivdual DomU partitions, and say both Dom0 and DomU
   have a partition formated ext3 with same label, then mount by label
   in Dom0 may end up picking up DomU''s partition by accident.
   In addition tools in DomU may well get confuse by seeing
''xdva1''
   without a corresponding ''xvda''.
   

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|