Xen users - Oct 2012 - "Boot loader did not return any data" to make HVM to PV

Hello
I''m trying to convert a HVM to PV dom0. The domU is a 10SP2 SLED (
Kernel 2.6.16.60-0.21  i386) and is a SLES11SP2 dom0 (Kernel
3.0.13-0.27-xen 64bit).
The truth is that every tutorial I see it differently and I do not
know where this error.
When I start the domU, pvgrub shows the options but when I select the
kenel-xen, returns the error "Boot loader did not return any data!"

The truth is that no more look, I''m lost in the different ways of
doing HVM to PV.

Below the log shipping configuration files

/--- domu.conf:
name = "ajrPrueba_pv"
maxmem = 300
memory = 300
bootloader="/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "destroy"
disk = [
       
"tap:qcow2:/mnt/xendomain/ajrPrueba_pv/SLED10_SP2_i586_disco3,xvda,w!"
       
,"tap:qcow2:/mnt/xendomain/ajrPrueba_pv/SLED10_SP2_i586_disco1,xvdb,w"
        ]
\--- domu.conf:


/--xend.log file when I start the domU
	[2012-10-18 12:25:52 3511] DEBUG (XendDomainInfo:103)
XendDomainInfo.create([''vm'', [''name'',
''ajrPrueba_pv''], [''memory'',
300], [''maxmem'', 300], [''on_poweroff'',
''destroy''], [''on_reboot'',
''restart''], [''on_crash'',
''destroy''], [''on_xend_start'',
''ignore''],
[''on_xend_stop'', ''ignore''],
[''vcpus'', 2], [''oos'', 1],
[''bootloader'',
''/usr/bin/pygrub''], [''bootloader_args'',
''''], [''image'', [''linux'',
[''videoram'', 4], [''tsc_mode'', 0],
[''nomigrate'', 0]]], [''s3_integrity'',
1], [''device'', [''tap'',
[''uname'',
''tap:qcow2:/mnt/xendomain/ajrPrueba_pv/SLED10_SP2_i586_disco3''],
[''dev'', ''xvda''], [''mode'',
''w!'']]], [''device'',
[''tap'', [''uname'',
''tap:qcow2:/mnt/xendomain/ajrPrueba_pv/SLED10_SP2_i586_disco1''],
[''dev'', ''xvdb''], [''mode'',
''w'']]]])
	[2012-10-18 12:25:52 3511] DEBUG (XendDomainInfo:2562)
XendDomainInfo.constructDomain
	[2012-10-18 12:25:52 3511] DEBUG (balloon:206) Balloon: 308284 KiB
free; need 16384; done.
	[2012-10-18 12:25:52 3511] DEBUG (XendDomain:482) Adding Domain: 16
	[2012-10-18 12:25:52 3511] DEBUG (XendDomainInfo:2903)
XendDomainInfo.initDomain: 16 256
	[2012-10-18 12:25:52 3511] INFO (XendDomainInfo:3358) Mounting
/mnt/xendomain/ajrPrueba_pv/SLED10_SP2_i586_disco3 on /dev/xvdz.
	[2012-10-18 12:25:52 3511] WARNING (BlktapController:169) WARNING:
using deprecated blktap module
	[2012-10-18 12:25:52 3511] DEBUG (DevController:95) DevController:
writing {''virtual-device-ext'': ''268441856'',
''backend-id'': ''0'',
''state'': ''1'',
''device-type'': ''disk'',
''backend'':
''/local/domain/0/backend/tap/0/268441856''} to
/local/domain/0/device/vbd/268441856.
	[2012-10-18 12:25:52 3511] DEBUG (DevController:97) DevController:
writing {''domain'': ''Domain-0'',
''frontend'':
''/local/domain/0/device/vbd/268441856'',
''uuid'':
''5074f361-967e-0936-5748-11c3bb7165c9'',
''bootable'': ''0'', ''dev'':
''/dev/xvdz'', ''state'': ''1'',
''params'':
''qcow2:/mnt/xendomain/ajrPrueba_pv/SLED10_SP2_i586_disco3'',
''mode'':
''w'', ''online'': ''1'',
''frontend-id'': ''0'',
''type'': ''tap''} to
/local/domain/0/backend/tap/0/268441856.
	[2012-10-18 12:25:52 3511] DEBUG (DevController:144) Waiting for 268441856.
	[2012-10-18 12:25:52 3511] DEBUG (DevController:671)
hotplugStatusCallback
/local/domain/0/backend/tap/0/268441856/hotplug-status.
	[2012-10-18 12:25:52 3511] DEBUG (DevController:685) hotplugStatusCallback 1.
	[2012-10-18 12:25:53 3511] DEBUG (DevController:616)
frontendStatusCallback /local/domain/0/device/vbd/268441856/state = 4
	[2012-10-18 12:25:53 27501] DEBUG (XendBootloader:130) Launching
bootloader as [''/usr/bin/pygrub'',
''--output=/var/run/xend/boot/xenbl.3769'',
''/dev/xvdz'']
+++
this is where the menu shows pvgrub (/ dev/xvdz1 is mountable and
contains the root of linux)
+++
	[2012-10-18 12:27:31 3511] ERROR (XendBootloader:231) Boot loader
didn''t return any data!
	[2012-10-18 12:27:31 3511] INFO (XendDomainInfo:3383) Unmounting
/dev/xvdz from /dev/xvdz.
	[2012-10-18 12:27:31 3511] DEBUG (XendDomainInfo:1278)
XendDomainInfo.destroyDevice: deviceClass = tap, device = /dev/xvdz
	[2012-10-18 12:27:31 3511] DEBUG (DevController:183) Waiting for
/dev/xvdz - destroyDevice.
	[2012-10-18 12:27:31 3511] DEBUG (DevController:692)
deviceDestroyCallback
/local/domain/0/backend/tap/0/268441856/hotplug-status.
	[2012-10-18 12:27:32 3511] DEBUG (DevController:692)
deviceDestroyCallback
/local/domain/0/backend/tap/0/268441856/hotplug-status.
	[2012-10-18 12:27:32 3511] DEBUG (DevController:701) deviceDestroyCallback 6.
	[2012-10-18 12:27:32 3511] ERROR (XendConfig:1211) dumping sxp from
device controllers
	Traceback (most recent call last):
	  File "/usr/lib64/python2.6/site-packages/xen/xend/XendConfig.py",
line 1197, in to_sxp
	    configs = controller.configurations(txn)
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py",
line 245, in configurations
	    return map(lambda x: self.configuration(x, transaction),
self.deviceIDs(transaction))
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py",
line 245, in <lambda>
	    return map(lambda x: self.configuration(x, transaction),
self.deviceIDs(transaction))
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py",
line 252, in configuration
	    configDict = self.getDeviceConfiguration(devid, transaction)
	  File "/usr/lib64/python2.6/site-packages/xen/xend/server/blkif.py",
line 164, in getDeviceConfiguration
	    ''bootable'')
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/server/DevController.py",
line 450, in readBackendTxn
	    raise VmError("Device %s not connected" % devid)
	VmError: Device 268441856 not connected
	[2012-10-18 12:27:32 3511] ERROR (XendDomainInfo:489) VM start failed
	Traceback (most recent call last):
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 475, in start
	    XendTask.log_progress(31, 60, self._initDomain)
	  File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py",
line 209, in log_progress
	    retval = func(*args, **kwds)
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 2905, in _initDomain
	    self._configureBootloader()
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 3379, in _configureBootloader
	    bootloader_args, kernel, ramdisk, args)
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/XendBootloader.py",
line 232, in bootloader
	    raise VmError, msg
	VmError: Boot loader didn''t return any data!
	[2012-10-18 12:27:32 3511] DEBUG (XendDomainInfo:3150)
XendDomainInfo.destroy: domid=16
	[2012-10-18 12:27:32 3511] DEBUG (XendDomainInfo:2470) No device model
	[2012-10-18 12:27:32 3511] DEBUG (XendDomainInfo:2472) Releasing devices
	[2012-10-18 12:27:32 3511] ERROR (XendDomainInfo:108) Domain
construction failed
	Traceback (most recent call last):
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 106, in create
	    vm.start()
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 475, in start
	    XendTask.log_progress(31, 60, self._initDomain)
	  File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py",
line 209, in log_progress
	    retval = func(*args, **kwds)
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 2905, in _initDomain
	    self._configureBootloader()
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py",
line 3379, in _configureBootloader
	    bootloader_args, kernel, ramdisk, args)
	  File
"/usr/lib64/python2.6/site-packages/xen/xend/XendBootloader.py",
line 232, in bootloader
	    raise VmError, msg
	VmError: Boot loader didn''t return any data!
\--xend.log file when I start the domU

/-- domU /grub/menu.list
title Xen -- SUSE Linux Enterprise Desktop 10 SP2 xen
    root (hd0,0)
    kernel /boot/xen.gz
    module /boot/vmlinuz-2.6.16.60-0.85.1-xen root=/dev/xvda1
vga=0x317 noresume splash=silent showopts clocksource=jiffies
console=xvc0
    module /boot/initrd-2.6.16.60-0.85.1-xen
\-- domU /grub/menu.list

/-- mkinitrd configuration
Kernel image:	/boot/vmlinuz-2.6.16.60-0.85.1-xen
Initrd image:	/boot/initrd-2.6.16.60-0.85.1-xen
Adding the root filesystem module (ext3)
Shared libs:	lib/ld-2.4.so lib/libacl.so.1.1.0 lib/libattr.so.1.1.0
lib/libblkid.so.1.0 lib/libc-2.4.so lib/libcom_err.so.2.1
lib/libdl-2.4.so lib/libext2fs.so.2.4 lib/libhistory.so.5.1
lib/libncurses.so.5.5 lib/libpthread-2.4.so lib/libreadline.so.5.1
lib/librt-2.4.so lib/libuuid.so.1.2 lib/libnss_files-2.4.so
lib/libnss_files.so.2 lib/libgcc_s.so.1

Driver modules:	ide-core ide-disk xenblk piix

Xen domU modules:	xennet xenblk
Filesystem modules:	jfs jbd ext3
Including:	initramfs fsck.ext3
No bootsplash for kernel flavor xen
\-- mkinitrd configuration


/-- fstab
/dev/xvda1 /                  ext3       acl,user_xattr        1 1
/dev/xvdb1 swap                 swap       defaults              0 0
proc                 /proc                proc       defaults              0 0
sysfs                /sys                 sysfs      noauto                0 0
debugfs              /sys/kernel/debug    debugfs    noauto                0 0
usbfs                /proc/bus/usb        usbfs      noauto                0 0
devpts               /dev/pts             devpts     mode=0620,gid=5       0 0
\-- fstab

On 10/19/2012 8:39 AM, Flako wrote:
>
> /-- domU /grub/menu.list
> title Xen -- SUSE Linux Enterprise Desktop 10 SP2 xen
>      root (hd0,0)
>      kernel /boot/xen.gz
>      module /boot/vmlinuz-2.6.16.60-0.85.1-xen root=/dev/xvda1
> vga=0x317 noresume splash=silent showopts clocksource=jiffies
> console=xvc0
>      module /boot/initrd-2.6.16.60-0.85.1-xen
> \-- domU /grub/menu.list
>
>
I think part of your issue is your menu.lst. Try adding another menu 
that looks like this and see what happens:

title Xen -- TEST SUSE Linux Enterprise Desktop 10 SP2 xen
     root (hd0,0)
     kernel /boot/vmlinuz-2.6.16.60-0.85.1-xen root=/dev/xvda1 vga=0x317 
noresume splash=silent showopts clocksource=jiffies console=xvc0
     initrd /boot/initrd-2.6.16.60-0.85.1-xen

HTH,
James

El 19/10/12 07:39, Flako escribió:
> The truth is that every tutorial I see it differently and I do not
> know where this error.
Which one you followed?
> When I start the domU, pvgrub shows the options but when I select the
> kenel-xen, returns the error "Boot loader did not return any
data!"
You can test pygrub manually:

# touch /var/run/xend/boot/xenbl.bogus
# /usr/lib/xen-default/bin/pygrub  \
  ''--output=/var/run/xend/boot/xenbl.bogus'' \
  ''/mnt/xendomain/ajrPrueba_pv/SLED10_SP2_i586_disco3''
# cat /var/run/xend/boot/xenbl.bogus
(see the path of extracted kernel and initrd images, check if they 
really was extracted. Delete them manually aftewards)
# rm /var/run/xend/boot/xenbl.bogus
("rm /var/run/xend/boot/*" might work, but check first)

pygrub will look for kernel and menu.list on the first image mentioned 
in the config. In your case it''s SLED10_SP2_i586_disco3. Change it if 
your boot device is the other one.

Note that you will probably need to move your disk image(s) from a 
partitioned block (or loop) device to a raw one. I would use kpartx and 
dd for that, but i have never done that with "tap:qcow2" backends. Use
with care. Maybe I''m outdated on this.
> The truth is that no more look, I''m lost in the different ways of
> doing HVM to PV.
A common issue is the attempt to mix instructions from different 
tutorials (they might use different approach, incompatible to each 
other). The other side is that it''s the way to get the understanding of
what''s going on, instead of blind execution. If this is your case, make
sure you know what''s you are doing.
> /--xend.log file when I start the domU
> [...]
> 	[2012-10-18 12:25:53 27501] DEBUG (XendBootloader:130) Launching
> bootloader as [''/usr/bin/pygrub'',
> ''--output=/var/run/xend/boot/xenbl.3769'',
''/dev/xvdz'']
> +++
> this is where the menu shows pvgrub (/ dev/xvdz1 is mountable and
> contains the root of linux)
> +++
> 	[2012-10-18 12:27:31 3511] ERROR (XendBootloader:231) Boot loader
> didn''t return any data!
It seems like pygrub was able to retrieve menu.list file, but not the 
boot image. See below for comment about xen.gz.
> /-- domU /grub/menu.list
> title Xen -- SUSE Linux Enterprise Desktop 10 SP2 xen
>      root (hd0,0)
>      kernel /boot/xen.gz
>      module /boot/vmlinuz-2.6.16.60-0.85.1-xen root=/dev/xvda1
> vga=0x317 noresume splash=silent showopts clocksource=jiffies
> console=xvc0
>      module /boot/initrd-2.6.16.60-0.85.1-xen
> \-- domU /grub/menu.list
I agree with James Pifer.
xen.gz is the bootable image of the hypervisor itself. It is executed on 
the host baremetal system once, and immediately boots Dom0. I has 
nothing to do in DomU image.

DomU''s menu.list should look very much similar to a normal
"xenless"
machine (or HVM guest), except that it will specify a xen enabled kernel 
(stored within DomU'' disk image) and a xen style root device. The
initrd
shall include modules for xenblk and xennet.

In resume, fixing menu.list might do the trick. Also, consider moving to 
a more "native" storage backend, instead of using "tap:qcow"
compatibility wrapper.

Greetings.

-- 
Alexandre Kouznetsov

2012/10/19 Alexandre Kouznetsov <alk@ondore.com>:
> El 19/10/12 07:39, Flako escribió:
>
>> The truth is that every tutorial I see it differently and I do not
>> know where this error.
>
> Which one you followed?
>
>
>> When I start the domU, pvgrub shows the options but when I select the
>> kenel-xen, returns the error "Boot loader did not return any
data!"
>
> You can test pygrub manually:
>
> # touch /var/run/xend/boot/xenbl.bogus
> # /usr/lib/xen-default/bin/pygrub  \
>  ''--output=/var/run/xend/boot/xenbl.bogus'' \
>  ''/mnt/xendomain/ajrPrueba_pv/SLED10_SP2_i586_disco3''
> # cat /var/run/xend/boot/xenbl.bogus
> (see the path of extracted kernel and initrd images, check if they really
> was extracted. Delete them manually aftewards)
> # rm /var/run/xend/boot/xenbl.bogus
> ("rm /var/run/xend/boot/*" might work, but check first)
>
> pygrub will look for kernel and menu.list on the first image mentioned in
> the config. In your case it''s SLED10_SP2_i586_disco3. Change it if
your boot
> device is the other one.
>
> Note that you will probably need to move your disk image(s) from a
> partitioned block (or loop) device to a raw one. I would use kpartx and dd
> for that, but i have never done that with "tap:qcow2" backends.
Use with
> care. Maybe I''m outdated on this.
>
>
>> The truth is that no more look, I''m lost in the different ways
of
>> doing HVM to PV.
>
> A common issue is the attempt to mix instructions from different tutorials
> (they might use different approach, incompatible to each other). The other
> side is that it''s the way to get the understanding of
what''s going on,
> instead of blind execution. If this is your case, make sure you know
what''s
> you are doing.
>
>> /--xend.log file when I start the domU
>> [...]
>>
>>         [2012-10-18 12:25:53 27501] DEBUG (XendBootloader:130)
Launching
>> bootloader as [''/usr/bin/pygrub'',
>> ''--output=/var/run/xend/boot/xenbl.3769'',
''/dev/xvdz'']
>> +++
>> this is where the menu shows pvgrub (/ dev/xvdz1 is mountable and
>> contains the root of linux)
>> +++
>>         [2012-10-18 12:27:31 3511] ERROR (XendBootloader:231) Boot
loader
>> didn''t return any data!
>
> It seems like pygrub was able to retrieve menu.list file, but not the boot
> image. See below for comment about xen.gz.
>
>
>> /-- domU /grub/menu.list
>> title Xen -- SUSE Linux Enterprise Desktop 10 SP2 xen
>>      root (hd0,0)
>>      kernel /boot/xen.gz
>>      module /boot/vmlinuz-2.6.16.60-0.85.1-xen root=/dev/xvda1
>> vga=0x317 noresume splash=silent showopts clocksource=jiffies
>> console=xvc0
>>      module /boot/initrd-2.6.16.60-0.85.1-xen
>> \-- domU /grub/menu.list
>
> I agree with James Pifer.
> xen.gz is the bootable image of the hypervisor itself. It is executed on
the
> host baremetal system once, and immediately boots Dom0. I has nothing to do
> in DomU image.
>
> DomU''s menu.list should look very much similar to a normal
"xenless" machine
> (or HVM guest), except that it will specify a xen enabled kernel (stored
> within DomU'' disk image) and a xen style root device. The initrd
shall
> include modules for xenblk and xennet.
>
> In resume, fixing menu.list might do the trick. Also, consider moving to a
> more "native" storage backend, instead of using
"tap:qcow" compatibility
> wrapper.
>
James:
   With the correction of menu.list worked (thanks)

Alexandre:
  What you say is true about following tutorials, I try to follow one
that "works" :). (sometimes it is difficult to follow when they are
based on different Linux)

   It pygrub manually running seems not to work because they qcow2
disk (as you say), I see from log xen called as:

DEBUG (XendBootloader: 130) Launching bootloader as
[''/usr/bin/pygrub'', ''- output = /
var/run/xend/boot/xenbl.8748'', ''/
dev / xvdz''].

Where /dev/xvdz what genre previously associated
SLED10_SP2_i586_disco3 disk. (I miss xvdz generate manually for
testing)

When you say "consider moving to a more ''native'' storage
backend" What
do you mean?
What is the most native backend?

Normally use tap: qcow2 by the reduction of disk space, is slower than
raw but for basic use of linux root I think is acceptable.
For intensive use (database or file server) use disk phy+lvm
Bad idea to combine qcow2 phy and the way I''m doing? which option
would be better?

Thank you both.

On Wed, Oct 24, 2012 at 6:27 AM, Flako <subforos@gmail.com>
wrote:
> Normally use tap: qcow2 by the reduction of disk space, is slower than
> raw but for basic use of linux root I think is acceptable.
> For intensive use (database or file server) use disk phy+lvm
> Bad idea to combine qcow2 phy and the way I''m doing? which option
> would be better?
Hehehe. I gave up on trying to make qcow2 (or vhd) work a long time
ago, due to the fact that there always seem to be some missing bits
that prevent them from working the way I want to.

That''s why I just stick with zfsonlinux :D

With that:
- you can use a sparse zvol with snapshot/clone to get pretty much all
the benefits of qcow2
- you can still use phy: with zvol.
- you can get compression on the lower storage layer

Another alternative would be to use sparse files on btrfs. It doesn''t
have emulated volumes (so you need tap: with sparse files), but other
than that (from xen''s perspective) it''s pretty much similar to
ZoL.

-- 
Fajar

Hi.

El 23/10/12 18:27, Flako escribió:
>    What you say is true about following tutorials, I try to follow one
> that "works" :). (sometimes it is difficult to follow when they
are
> based on different Linux)
Well, I expected a URL, to take a look on it...
> When you say "consider moving to a more ''native''
storage backend" What
> do you mean?
> What is the most native backend?
I guess "raw" (file, or device like LVM''s volume) is a better
term then
"native". It allows you to do all sort of things with the storage 
outside of your DomU, like snapshots for backups, access to individual 
partitions, pygrub...
> Normally use tap: qcow2 by the reduction of disk space, is slower than
> raw but for basic use of linux root I think is acceptable.
I just searched for some more reference, and found that apparently 
pygrub does not supports qcow disk images:
https://wiki.heprc.uvic.ca/twiki/bin/view/Main/QCOWSupport
(they have expired SSL cert).
Maybe that''s the root of your troubles.
> For intensive use (database or file server) use disk phy+lvm
> Bad idea to combine qcow2 phy and the way I''m doing? which option
> would be better?
I use raw "phy" storage type.

Never tried to combine different storage types (actually, only worked 
with "file" and "phy" over lvm), but it seems to be a good
idea to
maintain this things homogeneous unless there is a *good* reason not to 
do so.
The root FS on my typical Linux PV instance takes less then 2GB. Even if 
a qcow could cut that to the half, I believe it does not worth it.

I guess there are tools to convert from qcow2 to raw image.

-- 
Alexandre Kouznetsov

2012/10/23 Alexandre Kouznetsov <alk@ondore.com>:
> Hi.
>
> El 23/10/12 18:27, Flako escribió:
>
>>    What you say is true about following tutorials, I try to follow one
>> that "works" :). (sometimes it is difficult to follow when
they are
>> based on different Linux)
>
> Well, I expected a URL, to take a look on it...
>
>
>> When you say "consider moving to a more ''native''
storage backend" What
>> do you mean?
>> What is the most native backend?
>
> I guess "raw" (file, or device like LVM''s volume) is a
better term then
> "native". It allows you to do all sort of things with the storage
outside of
> your DomU, like snapshots for backups, access to individual partitions,
> pygrub...
>
>
>> Normally use tap: qcow2 by the reduction of disk space, is slower than
>> raw but for basic use of linux root I think is acceptable.
>
> I just searched for some more reference, and found that apparently pygrub
> does not supports qcow disk images:
> https://wiki.heprc.uvic.ca/twiki/bin/view/Main/QCOWSupport
> (they have expired SSL cert).
> Maybe that''s the root of your troubles.
>
>
>> For intensive use (database or file server) use disk phy+lvm
>> Bad idea to combine qcow2 phy and the way I''m doing? which
option
>> would be better?
>
> I use raw "phy" storage type.
>
> Never tried to combine different storage types (actually, only worked with
> "file" and "phy" over lvm), but it seems to be a good
idea to maintain this
> things homogeneous unless there is a *good* reason not to do so.
> The root FS on my typical Linux PV instance takes less then 2GB. Even if a
> qcow could cut that to the half, I believe it does not worth it.
>
> I guess there are tools to convert from qcow2 to raw image.
>
>
> --
> Alexandre Kouznetsov
>
Hello
Use ZFS is tempting, but not natively supported by SUSE SLES (I have to wait ..)
A btrfs I have to try .. :)

Greetings.