Xen users - Dec 2007 - Problems converting FC8 HVM guest to PV

I''m running a Fedora 8 Dom0, x86_64, with a Fedora 8 DomU, HVM, i386. 
I''d like to convert the (working) DomU to PV, but I''ve been
banging my
head against a brick wall for the last few hours. Can anyone point me in 
the right direction?

Plan A was to use pygrub, and to make a new DomU ramdisk with the 
xenblock module. I ran mkinitrd with --preload=xenblock, and set up a 
new grub entry on the DomU:

title Fedora (2.6.21-2952.fc8xen.xenblk)
  root (hd0,0)
  kernel /xen.gz-2.6.21-2952.fc8
  module /vmlinuz-2.6.21-2952.fc8xen ro root=LABEL=/ rhgb quiet
  module /initrd-2.6.21-2952.fc8xen.xenblk.img

The config file on the Dom0 was:

name   = "fc8-32-001"
memory = 1024
vcpus  = 1
vif  = [ ''mac=00:0f:b5:ae:4f:2a, bridge=xenbr0'' ]
disk = [ ''phy:/dev/VG_Guests/FC8-32-001,hda,w'' ,
          ''file:/tmp/FC8-32.ISO,hdc:cdrom,r''];
bootloader="/usr/bin/pygrub"
boot = "cd"
sdl  = 1
vnc  = 0

When I created the domU, pygrub correctly gave me the boot menu from the 
DomU, but it wouldn''t boot the entry above; it complained that it
wasn''t
a valid kernel ("not a Xen-ELF image", and it wasn''t a
xen_guest).
However, it *is* a valid (gzipped) Xen kernel (I know because readelf 
reports a __xen_guest entry, and if I use it as the HVM DomU kernel then 
uname reports it as a Xen kernel). So, I figured that pygrub was broken, 
and moved on to plan B.

Plan B was to move the 32-bit guest kernel and ramdisk to the Dom0, and 
modify the config file. I''ve tried lots of different variations on the 
config file, and I''ve currently got:

name   = "fc8-32-002"
memory = 1024
vcpus  = 1
vif  = [ ''mac=00:0f:b5:ae:4f:2a, bridge=xenbr0'' ]
disk = [ ''phy:/dev/VG_Guests/FC8-32-001,hda,w'' ];
#bootloader="/usr/bin/pygrub"
kernel  = "/boot/fc8-32/vmlinuz-2.6.21-2952.fc8xen"
ramdisk = "/boot/fc8-32/initrd-2.6.21-2952.fc8xen.img"
root    = "/dev/hda ro"
sdl     = 1
vnc     = 0

This doesn''t work, and none of the variations worked. I copied the 
32-bit DomU Xenified kernel into the Dom0 path in the ''kernel''
line.
I''ve tried the config with and without the ramdisk line, and with the 
original 32-bit guest initrd, as well as my new xenblock version. I''ve 
tried variations on the ''hda'' as well, and on the root line. I
get
several different error messages.

Any ideas? I suspect that I''m missing something obvious.

Thanks -

Evan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thursday 27 December 2007 06:35:54 pm Evan Lavelle
wrote:
> title Fedora (2.6.21-2952.fc8xen.xenblk)
>   root (hd0,0)
>   kernel /xen.gz-2.6.21-2952.fc8
>   module /vmlinuz-2.6.21-2952.fc8xen ro root=LABEL=/ rhgb quiet
>   module /initrd-2.6.21-2952.fc8xen.xenblk.img
Since it''s likely to be slow on the list now, I''ll give this a
stab, tho'' I''ll
warn you I haven''t actually done this myself. However, some things
stand out:

The reason Plan A didn''t work is you can''t load the xen
hypervisor (xen.gz)
inside a domu. Try deleting the ''kernel /xen.gz...'' line and 
change ''module /vmlinuz...'' to ''kernel
/vmlinuz...''. I personally would use
the original fc8 /initrd-2.6.21-2952.fc8xen.img. At a minimum, a custom 
initrd also has to have ''--preload=xennet''.

The basic problem with your other plans is it looks like the hvm install 
created a whole disk (hda) with an fc8 partition on it in, presumably hda1.
I don''t know if specifying hda in the disk= parm and
root=''hda1 ro'' will work
in Plan B, but it''s closer to what you mean.

If you get something to work, presumably with the help of someone wiser than 
me ;-), you will also want to eventually replace hda* with xvda*. Good luck, 
and keep us posted.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, 2007-12-27 at 19:57 -0500, jim burns wrote:
> On Thursday 27 December 2007 06:35:54 pm Evan Lavelle wrote:
> > title Fedora (2.6.21-2952.fc8xen.xenblk)
> >   root (hd0,0)
> >   kernel /xen.gz-2.6.21-2952.fc8
> >   module /vmlinuz-2.6.21-2952.fc8xen ro root=LABEL=/ rhgb quiet
> >   module /initrd-2.6.21-2952.fc8xen.xenblk.img
> 
> Since it''s likely to be slow on the list now, I''ll give
this a stab, tho'' I''ll
> warn you I haven''t actually done this myself. However, some things
stand out:
> 
> The reason Plan A didn''t work is you can''t load the xen
hypervisor (xen.gz)
> inside a domu. Try deleting the ''kernel /xen.gz...'' line
and
> change ''module /vmlinuz...'' to ''kernel
/vmlinuz...''. I personally would use
> the original fc8 /initrd-2.6.21-2952.fc8xen.img. At a minimum, a custom 
> initrd also has to have ''--preload=xennet''.

I don''t have my notes in front of me, but I believe Jim pretty much
nailed it (at least with option A).
I''ve done several HVM -> PV conversions and always use the first
option
you described.
For fedora it comes down to: 
yum install kernel-xen
mkinitrd --preload=xenblk --preload=xennet
(if you don''t have xennet your domU will still work, but you
won''t have
a network at boot, you can modprobe xennet after boot if you forget)

Modify the grub config of the domU:
  title Fedora (2.6.21-2952.fc8xen.xenblk)
    root (hd0,0)
    kernel /vmlinuz-2.6.21-2952.fc8xen ro root=LABEL=/ rhgb quiet
    initrd /initrd-2.6.21-2952.fc8xen.xenblk.img

And switch the config file to use pygrub

If you have any issues, just post back to the list.  I''ll be able to
get
my notes tomorrow.


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Thanks guys - the good news is that I can boot the PV DomU with your fixes:

title Fedora (PV boot, Xen kernel, 2952)
	root (hd0,0)
	kernel /vmlinuz-2.6.21-2952.fc8xen ro root=LABEL=/ rhgb quiet
	initrd /initrd-2.6.21-2952.fc8xen.pv.img

name   = "fc8-32-001"
memory = 1024
vcpus  = 1
vif  = [ ''mac=00:0f:b5:ae:4f:2a, bridge=xenbr0'' ]
disk = [ ''phy:/dev/VG_Guests/FC8-32-001,hda,w''  ];
bootloader="/usr/bin/pygrub"
boot = "cd"
sdl  = 1
vnc  = 0

I got rid of the root line with no problems. I couldn''t boot at all
with
the original ramdisk (I got a "mount: could not find filesystem 
''dev/root''" error), but had no problem with the xenblk
and
xenblock/xennet ramdisks.

The *bad* news is, well, bad. The DomU appears to be Ok at first sight, 
but crashes when I run a specific program (wine, which is a 
Windows-on-Unix emulator [yes, I know that''s crazy, but I need it for 
testing]). I get a kernel bug:

kernel BUG at include/linux/tracehook.h:369!
invalid opcode: 0000 [#1]

Should I bother reporting this? The Dom0 appears flakey after this so I 
need to restart the entire machine.

I could live with this, by going back to the HVM guest, but this is 
non-trivial. The PV boot messed up various things, and an HVM create 
leaves me locked out of the DomU:

1 - The console doesn''t work (but it never has on my HVM
DomU''s; I get a
  "xenconsole: could not read tty from store" error, with nothing
useful
in the qemu log)

2 - networking isn''t started; I get the normal eth0 startup failed 
message, but there''s no subsequent attempt to startup eth1/2/etc

3 - The sdl window hangs up at the anacron boot message, which is normal 
on the FC8 Dom0 (after FC8 messes up the serial console and inittab).

I''m slowly working my way through this lot, with some (Ok, major) 
irritation. FC8 (or Xen? I don''t know) replaced the DomU inittab with 
one it obviously thought was better, which left the DomU in run level 3. 
fixing that has helped. Network next....

:(

Evan

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Evan Lavelle wrote:
> Thanks guys - the good news is that I can boot the PV DomU with your 
> fixes:
>
> title Fedora (PV boot, Xen kernel, 2952)
>     root (hd0,0)
>     kernel /vmlinuz-2.6.21-2952.fc8xen ro root=LABEL=/ rhgb quiet
>     initrd /initrd-2.6.21-2952.fc8xen.pv.img
>
> name   = "fc8-32-001"
> memory = 1024
> vcpus  = 1
> vif  = [ ''mac=00:0f:b5:ae:4f:2a, bridge=xenbr0'' ]
> disk = [ ''phy:/dev/VG_Guests/FC8-32-001,hda,w''  ];
> bootloader="/usr/bin/pygrub"
> boot = "cd"
> sdl  = 1
> vnc  = 0
>
> I got rid of the root line with no problems. I couldn''t boot at
all
> with the original ramdisk (I got a "mount: could not find filesystem 
> ''dev/root''" error), but had no problem with the
xenblk and
> xenblock/xennet ramdisks.
>
> The *bad* news is, well, bad. The DomU appears to be Ok at first 
> sight, but crashes when I run a specific program (wine, which is a 
> Windows-on-Unix emulator [yes, I know that''s crazy, but I need it
for
> testing]). I get a kernel bug:
>
> kernel BUG at include/linux/tracehook.h:369!
> invalid opcode: 0000 [#1]
>
> Should I bother reporting this? The Dom0 appears flakey after this so 
> I need to restart the entire machine.
Can you do a fast, clean FC8 PV install and test Wine on it?

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Friday 28 December 2007 09:03:39  Evan Lavelle wrote:
> The *bad* news is, well, bad. The DomU appears to be Ok at first sight,
> but crashes when I run a specific program (wine, which is a
> Windows-on-Unix emulator [yes, I know that''s crazy, but I need it
for
> testing]). I get a kernel bug:
Wine is one of those badly behaved glibc negative segment offset programs. 
It''s painful enough waiting for it to load in dom0. I''d hate
to see what it
does in a domu. Still, if you have time to follow Nico''s suggestion,
might be
interesting.
> 1 - The console doesn''t work (but it never has on my HVM
DomU''s; I get a
>   "xenconsole: could not read tty from store" error, with nothing
useful
> in the qemu log)
If you want to debug this further, post your hvm config. The way sdl grabs the 
mouse pointer drives me crazy.
> 2 - networking isn''t started; I get the normal eth0 startup failed
> message, but there''s no subsequent attempt to startup eth1/2/etc
Look at ''brctl show''. You may have dom0''s eth0 as a
bridge, which fc8 (and any
modern 3.1.x xen) does, and may have to change your hvm vif''s bridge 
accordingly, at least as a test.
> 3 - The sdl window hangs up at the anacron boot message, which is normal 
> on the FC8 Dom0 (after FC8 messes up the serial console and inittab).
Somebody correct the syntax if this is wrong, but appending
''console=tty
console=xvc0'' to your domu''s grub''s kernel line may
help.

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nico Kadel-Garcia wrote:
> Can you do a fast, clean FC8 PV install
Ummm... no. Is that even possible? :)
> and test Wine on it?
I needed another FC8 PV guest anyway, so I gave this a go. I used 
virt-install from HTTP:

  Dom0 FC8 x86_64, kernel 2.6.21-2952
  DomU FC8 x86_64, kernel 2.6.21-2950, PV, on LVM
  Intel Core 2 Duo/6600

xm info reports Xen 3.1, though various log messages report 3.0.

I then installed Wine and the Windows app. The Windows installer ran 
fine under Wine, as well as the licensing. The app itself also runs Ok 
until it exits, at which point I get the crash. Kernel output attached 
(the output is actually for the crash of the i386 DomU that I converted 
from HVM, with kernel 2952, but it seems identical at first sight).

Evan


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> On Thursday 27 December 2007 06:35:54 pm Evan Lavelle wrote:
> > title Fedora (2.6.21-2952.fc8xen.xenblk)
> >   root (hd0,0)
> >   kernel /xen.gz-2.6.21-2952.fc8
> >   module /vmlinuz-2.6.21-2952.fc8xen ro root=LABEL=/ rhgb quiet
> >   module /initrd-2.6.21-2952.fc8xen.xenblk.img
>
> Since it''s likely to be slow on the list now, I''ll give
this a stab, tho''
> I''ll warn you I haven''t actually done this myself.
However, some things
> stand out:
>
> The reason Plan A didn''t work is you can''t load the xen
hypervisor (xen.gz)
> inside a domu. Try deleting the ''kernel /xen.gz...'' line
and
> change ''module /vmlinuz...'' to ''kernel
/vmlinuz...''. I personally would use
> the original fc8 /initrd-2.6.21-2952.fc8xen.img. At a minimum, a custom
> initrd also has to have ''--preload=xennet''.
Don''t you get some kind of Xennified initrd installed with the kernel
RPM?
I''d have thought Fedora would supply a sane default one?
> The basic problem with your other plans is it looks like the hvm install
> created a whole disk (hda) with an fc8 partition on it in, presumably hda1.
> I don''t know if specifying hda in the disk= parm and
root=''hda1 ro'' will
> work in Plan B, but it''s closer to what you mean.
Yeah, that works.  If you''re converting HVM->PV you''ll need
to still pass the
VBD through as a whole disk so that the guest can read its partition table.

If you need to mount partitions out of it, you can use the "lomount"
tool
that''s supplied with Xen (originally came from Qemu) or you can use
kpartx.
> If you get something to work, presumably with the help of someone wiser
> than me ;-), you will also want to eventually replace hda* with xvda*.
Yes, that''s true too.

Good advice!

Cheers,
Mark
> Good 
> luck, and keep us posted.
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users


-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> The *bad* news is, well, bad. The DomU appears to be Ok at first sight,
> but crashes when I run a specific program (wine, which is a
> Windows-on-Unix emulator [yes, I know that''s crazy, but I need it
for
> testing]). I get a kernel bug:
>
> kernel BUG at include/linux/tracehook.h:369!
> invalid opcode: 0000 [#1]
>
> Should I bother reporting this? The Dom0 appears flakey after this so I
> need to restart the entire machine.
I''m not entirely sure that this is intended to be supported, but it
certainly
shouldn''t cause the kernel to BUG out!  So yes, please, it''s
probably worth
posting this on the -devel list with a subject lines along the lines 
of "Reproduceable BUG oops in domU (plus subsequent dom0 instability)"
so
that it''s recorded.

There may be a similar bug in the Xen Bugzilla but that''s still not
used very
effectively these days :-(

What kind of instability do you see in dom0 after this error?  Do you have 
problems with other domUs too?
> I could live with this, by going back to the HVM guest, but this is
> non-trivial. The PV boot messed up various things, and an HVM create
> leaves me locked out of the DomU:
>
> 1 - The console doesn''t work (but it never has on my HVM
DomU''s; I get a
>   "xenconsole: could not read tty from store" error, with nothing
useful
> in the qemu log)
That''s odd.

You would have to configure the guest to output to the virtual serial port in 
any case, in order to get the xm console to work with an HVM guest. 
I''m not
sure if that''s why you''re having problems though; I remember
seeing that
error but not what caused it.
> 2 - networking isn''t started; I get the normal eth0 startup failed
> message, but there''s no subsequent attempt to startup eth1/2/etc
Hmmm.  Well this ought to be fixable...

If you try to manually ifup or ifconfig those interfaces, do they show up at 
all?

Maybe F8 has remove configuration for the virtual hardware which
"disappeared"
when you converted to PV.  It ought to be possible to reconfigure that 
hardware using system-config-network, etc, though...  Ideally there ought to 
be a way of making it retain the config so you can HVM/PV "dual boot".
I''m
not quite sure what''d be involved in terms of configuring Fedora there,
though.
> 3 - The sdl window hangs up at the anacron boot message, which is normal
> on the FC8 Dom0 (after FC8 messes up the serial console and inittab).
I''ve addressed this in another e-mail.
> I''m slowly working my way through this lot, with some (Ok, major)
> irritation. FC8 (or Xen? I don''t know) replaced the DomU inittab
with
> one it obviously thought was better, which left the DomU in run level 3.
> fixing that has helped. Network next....
That''s probably a F8 or virt-install thing.  virt-manager installs
graphical
guests by default; maybe there''s an option to virt-install you can
tweak to
change default runlevel?

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Wine is one of those badly behaved glibc negative segment offset programs.
> It''s painful enough waiting for it to load in dom0. I''d
hate to see what it
> does in a domu.
I guess it could be provoking a bug due to that behaviour.  I''m
inclined to
hazard a guess that it''s something to do with Wine''s use of
segments.  That''s
just a not-very-educated guess though, so I could be wrong there!
> If you want to debug this further, post your hvm config. The way sdl grabs
> the mouse pointer drives me crazy.
You could use VNC instead?  Although *not* having grabbing can make Linux 
guests awkward to deal with (Windows guests can use the USB tablet emulation 
to make grabless operation work sensibly).
> > 3 - The sdl window hangs up at the anacron boot message, which is
normal
> > on the FC8 Dom0 (after FC8 messes up the serial console and inittab).
>
> Somebody correct the syntax if this is wrong, but appending
''console=tty
> console=xvc0'' to your domu''s grub''s kernel line
may help.
The console=xvc0 won''t help for an HVM domain because xvc0
isn''t there; you''d
need it in a PV guest though.

console=tty ought to help get all appropriate boot output, I''d have
thought.

You might want to check inittab starts the correct gettys to give you a login 
on tty1 (e.g. that it''s not doing something silly like just starting a
getty
on /dev/xvc0 or /dev/ttyS0 or something like that).

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users