xen discuss - Jul 2009 - can''t boot 2009.06 domU on Xen 3.4.1 / CentOS 5.3 dom0

I''ve got a CentOS 5.3 dom0 with Xen 3.4.1-rc5 (or so). I''ve
tried the same stuff below with 3.4.0, no difference. I''m trying to
install 2009.06 PV domU based on instructions from [1] and [2]. I can run the
install fine, I can also get the kernel and boot archive (from [2]) after the
install. But for the life of me I can''t get the installed domU to boot.

If I try to use pygrub, which I think in 3.4.1 has the necessary changes to boot
ZFS, I get "bootloader did not return any data" straight away in the
xm create. This would indicate that something isn''t right about the
zpool or pygrub. If I use the kernel and boot archive (see [2]) then it does
start to boot but panics within seconds, see below for console output, which
also indicates that something in ZFS boot is not happy.

I''ve tried every combination I can think of for providing storage to
the domU, including LVM, file, and separate device. When using a separate device
I tried specifying the disk as following in the Xen config file:

  disk = [''phy:/dev/sdb,0,w'']

Also going to sdb1, or setting the target device to xvda, etc.. Doesn''t
seem to matter, still fails in pygrub or early boot.

Note that I can boot the LiveCD again and import the ZFS pool from disk (zpool
import -f rpool), it''s definitely there. I''ve tried this on
hardware (Intel 5500-based server) and under VMware, same failure.

Any ideas?

Thanks,
Josh

[1]:
http://bderzhavets.wordpress.com/2009/06/03/setup-opensolaris-2009-06-pv-domu-at-xen-3-5-unstable-dom0-kernel-2-6-30-rc6-tip/
[2]: http://blogs.sun.com/levon/entry/opensolaris_2009_06_guest_domain

--console dump mentioned above--

v3.4.1-rc5 chgset ''Mon Jun 29 11:16:43 2009 +0100
19676:8e78b81d67b9''
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.

panic[cpu0]/thread=fffffffffbc60560: assertion failed:
nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_TXG, &txg) == 0, file:
../../common/fs/zfs/spa.c, line: 2368

fffffffffbc93140 genunix:assfail+7e ()
fffffffffbc93190 zfs:spa_check_rootconf+8f ()
fffffffffbc93230 zfs:spa_get_rootconf+6c ()
fffffffffbc93270 zfs:spa_import_rootpool+25 ()
fffffffffbc932d0 zfs:zfs_mountroot+eb ()
fffffffffbc932f0 genunix:fsop_mountroot+1a ()
fffffffffbc93320 genunix:rootconf+12d ()
fffffffffbc93360 genunix:vfs_mountroot+63 ()
fffffffffbc93390 genunix:main+f0 ()
fffffffffbc933a0 unix:_locore_start+80 ()

skipping system dump - no dump device configured
rebooting...
-- 
This message posted from opensolaris.org

FYI here''s what I see when I boot the LiveCD and import rpool after an
install. In this case I rebooted after the install and unsuccessfully attempted
to boot the domU. Then I used my install config (with the CD and boot
kernel/microroot) to boot back to the LiveCD and inspect the disk.

--cut here--

<pre>
jack@opensolaris:~# zpool import rpool
cannot import ''rpool'': pool may be in use from other system,
it was last accessed by opensolaris (hostid: 0x32851) on Wed Jul  1 13:06:05
2009
use ''-f'' to import anyway
jack@opensolaris:~# zpool import -f rpool
jack@opensolaris:~# zpool status
  pool: rpool
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c7d0s0    ONLINE       0     0     0

errors: No known data errors
jack@opensolaris:~# zfs list 
NAME                        USED  AVAIL  REFER  MOUNTPOINT
rpool                      3.70G  6.08G    74K  /rpool
rpool/ROOT                 2.82G  6.08G    19K  legacy
rpool/ROOT/opensolaris     2.82G  6.08G  2.82G  /
rpool/dump                  384M  6.08G   384M  -
rpool/export               63.5K  6.08G    21K  /export
rpool/export/home          42.5K  6.08G    21K  /export/home
rpool/export/home/spectra  21.5K  6.08G  21.5K  /export/home/spectra
rpool/swap                  512M  6.43G   157M  -
</pre>
-- 
This message posted from opensolaris.org

Please, post 
1. your installation profile and shell commands to copy  solaris kernel and
microroot off the disk to Dom0.
2. uname -a
3. xm info
4. brctl show

--- On Wed, 7/1/09, Josh Carter <no-reply@opensolaris.org> wrote:

From: Josh Carter <no-reply@opensolaris.org>
Subject: [xen-discuss] can''t boot 2009.06 domU on Xen 3.4.1 / CentOS
5.3 dom0
To: xen-discuss@opensolaris.org
Date: Wednesday, July 1, 2009, 4:21 PM

I''ve got a CentOS 5.3 dom0 with Xen 3.4.1-rc5 (or so). I''ve
tried the same stuff below with 3.4.0, no difference. I''m trying to
install 2009.06 PV domU based on instructions from [1] and [2]. I can run the
install fine, I can also get the kernel and boot archive (from [2]) after the
install. But for the life of me I can''t get the installed domU to boot.

If I try to use pygrub, which I think in 3.4.1 has the necessary changes to boot
ZFS, I get "bootloader did not return any data" straight away in the
xm create. This would indicate that something isn''t right about the
zpool or pygrub. If I use the kernel and boot archive (see [2]) then it does
start to boot but panics within seconds, see below for console output, which
also indicates that something in ZFS boot is not happy.

I''ve tried every combination I can think of for providing storage to
the domU, including LVM, file, and separate device. When using a separate device
I tried specifying the disk as following in the Xen config file:

  disk = [''phy:/dev/sdb,0,w'']

Also going to sdb1, or setting the target device to xvda, etc.. Doesn''t
seem to matter, still fails in pygrub or early boot.

Note that I can boot the LiveCD again and import the ZFS pool from disk (zpool
import -f rpool), it''s definitely there. I''ve tried this on
hardware (Intel 5500-based server) and under VMware, same failure.

Any ideas?

Thanks,
Josh

[1]:
http://bderzhavets.wordpress.com/2009/06/03/setup-opensolaris-2009-06-pv-domu-at-xen-3-5-unstable-dom0-kernel-2-6-30-rc6-tip/
[2]: http://blogs.sun.com/levon/entry/opensolaris_2009_06_guest_domain

--console dump mentioned above--

v3.4.1-rc5 chgset ''Mon Jun 29 11:16:43 2009 +0100
19676:8e78b81d67b9''
SunOS Release 5.11 Version snv_111b 64-bit
Copyright 1983-2009 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.

panic[cpu0]/thread=fffffffffbc60560: assertion failed:
nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_TXG, &txg) == 0, file:
../../common/fs/zfs/spa.c, line: 2368

fffffffffbc93140 genunix:assfail+7e ()
fffffffffbc93190 zfs:spa_check_rootconf+8f ()
fffffffffbc93230 zfs:spa_get_rootconf+6c ()
fffffffffbc93270 zfs:spa_import_rootpool+25 ()
fffffffffbc932d0 zfs:zfs_mountroot+eb ()
fffffffffbc932f0 genunix:fsop_mountroot+1a ()
fffffffffbc93320 genunix:rootconf+12d ()
fffffffffbc93360 genunix:vfs_mountroot+63 ()
fffffffffbc93390 genunix:main+f0 ()
fffffffffbc933a0 unix:_locore_start+80 ()

skipping system dump - no dump device configured
rebooting...
-- 
This message posted from opensolaris.org
_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org

> Please, post
> 1. your installation profile and shell commands to copy & solaris
kernel and microroot off the disk to Dom0.
> 2. uname -a
> 3. xm info
> 4. brctl show
Ok, here goes:

1) Install and run profiles attached. Shell commands to copy kernel/boot archive
are verbatim from [1].

2) uname -a (dom 0): Linux bashful.localdomain 2.6.18.8-xen #1 SMP Mon Jun 29
09:21:28 MDT 2009 x86_64 x86_64 x86_64 GNU/Linux

3) xm info attached.

4) brctl show
bridge name	bridge id		STP enabled	interfaces
eth0		8000.001517348ead	no		peth0

Note that at the moment I''m turning off iptables in dom0 to get domU
networking to work, I haven''t yet changed iptables to allow the
traffic. Not relevant to the problem, anyway.

Also note that I did some more digging on the panic mentioned in my first note,
which led me to [2] that has the same stack trace. This problem was related to
grub, so I goofed around with updating the grub in my domU disk, but it had no
effect. Since I''m providing the kernel and ramdisk, I assume the
domU''s grub isn''t involved much (if at all) during domU boot.

Thanks for any help you can provide,
Josh

[1]: http://blogs.sun.com/levon/entry/opensolaris_2009_06_guest_domain
[2]: https://opensolaris.org/jive/message.jspa?messageID=309582
-- 
This message posted from opensolaris.org

Josh,

While you have the zpool imported, please check the version number.

zpool get all rpool | grep -i vers

This should be 14 (or less) which is what pygrub in 3.4 understands, but
I wanted to verify this.

You can find the version that pygrub supports by looking at
tools/libfsimage/zfs/zfs-include/zfs.h
and looking for SPA_VERSION.


Thanks,
Susan


Josh Carter wrote:
> FYI here''s what I see when I boot the LiveCD and import rpool
after an install. In this case I rebooted after the install and unsuccessfully
attempted to boot the domU. Then I used my install config (with the CD and boot
kernel/microroot) to boot back to the LiveCD and inspect the disk.
> 
> --cut here--
> 
> <pre>
> jack@opensolaris:~# zpool import rpool
> cannot import ''rpool'': pool may be in use from other
system, it was last accessed by opensolaris (hostid: 0x32851) on Wed Jul  1
13:06:05 2009
> use ''-f'' to import anyway
> jack@opensolaris:~# zpool import -f rpool
> jack@opensolaris:~# zpool status
>   pool: rpool
>  state: ONLINE
>  scrub: none requested
> config:
> 
>         NAME        STATE     READ WRITE CKSUM
>         rpool       ONLINE       0     0     0
>           c7d0s0    ONLINE       0     0     0
> 
> errors: No known data errors
> jack@opensolaris:~# zfs list 
> NAME                        USED  AVAIL  REFER  MOUNTPOINT
> rpool                      3.70G  6.08G    74K  /rpool
> rpool/ROOT                 2.82G  6.08G    19K  legacy
> rpool/ROOT/opensolaris     2.82G  6.08G  2.82G  /
> rpool/dump                  384M  6.08G   384M  -
> rpool/export               63.5K  6.08G    21K  /export
> rpool/export/home          42.5K  6.08G    21K  /export/home
> rpool/export/home/spectra  21.5K  6.08G  21.5K  /export/home/spectra
> rpool/swap                  512M  6.43G   157M  -
> </pre>

Hi Susan,

zpool version is 14, zfs version is 3. Grepping in the directory you mentioned
(in my dom0 xen-3.4 source tree), I find:

zfs.h:#define	SPA_VERSION			14ULL
zfs_znode.h:#define	ZPL_VERSION		3ULL

So it looks like pygrub *should* be able to boot the domU. Something''s
messed up with what''s on that disk, no matter if it''s pygrub
or a kernel from the dom0 that''s trying to access it. But then why does
the LiveCD import the pool just fine?

Thanks and best regards,
Josh
-- 
This message posted from opensolaris.org

Josh,

In your run_configuration.txt file try changing
   extra = "/platform/i86xpv/kernel/amd64/unix -B
zfs-bootfs=rpool/ROOT/opensolaris"
to
   extra = "/platform/i86xpv/kernel/amd64/unix -B
zfs-bootfs=rpool/ROOT/opensolaris,bootpath=/xpvd/xdf@51712:a"
since zfs needs the bootpath variable to boot.


Susan

>name = "storage"
>uuid = "f4b21d4b-f5c7-46a4-8a62-7765fbdb2cb0"
>vcpus = 1
>memory = "1536"
> kernel = "/var/xen/boot/opensolaris/kernel"
>ramdisk = "/var/xen/boot/opensolaris/ramdisk"
>extra = "/platform/i86xpv/kernel/amd64/unix -B
zfs-bootfs=rpool/ROOT/opensolaris"
I believe the mistake is in extra kernel boot command line.
Details here for  Nevada DomU with ZFS image :-

http://bderzhavets.wordpress.com/2008/12/08/install-snv103-pv-domu-with-zfs-image-at-xen-331-opensuse-111-dom0-all-64-bit/

Get the ZFS-ID of the /ROOT/snv_103 zpool. First, find out the pool_guid.


# zpool get guid rpool

# NAME PROPERTY VALUE SOURCE

rpool guid 17306096753167233998


************

Then run :-

************


bash-3.2# zdb -d -e 17306096753167233998

Dataset mos [META], ID 0, cr_txg 4, 1.74M, 131 objects

Dataset 17306096753167233998/swap [ZVOL], ID 37, cr_txg 15, 4.27M, 3 objects

Dataset 17306096753167233998/ROOT/snv_103 [ZPL], ID 57, cr_txg 31, 5.99G, 286127
objects



name = “SNV103PV”

vcpus = 1

memory = “2048″

kernel = “/etc/xen/nevada/unix”

ramdisk = “/etc/xen/nevada/boot_archive”

extra = “/platform/i86xpv/kernel/amd64/unix – nowin -B bootpath=/xpvd/xdf@0:a -B
console=xen -B zfs-bootfs=rpool/57″

disk = [''phy:/dev/sdb12,0,w'']

vif = [ ''bridge=br1'']

on_shutdown = “destroy”

on_reboot = “destroy”

on_crash = “destroy”

--- On Thu, 7/2/09, Josh Carter <no-reply@opensolaris.org> wrote:

From: Josh Carter <no-reply@opensolaris.org>
Subject: Re: [xen-discuss] can''t boot 2009.06 domU on Xen 3.4.1 /
CentOS 5.3 dom0
To: xen-discuss@opensolaris.org
Date: Thursday, July 2, 2009, 4:13 PM
> Please, post
> 1. your installation profile and shell commands to copy & solaris
kernel and microroot off the disk to Dom0.
> 2. uname -a
> 3. xm info
> 4. brctl show
Ok, here goes:

1) Install and run profiles attached. Shell commands to copy kernel/boot archive
are verbatim from [1].

2) uname -a (dom 0): Linux bashful.localdomain 2.6.18.8-xen #1 SMP Mon Jun 29
09:21:28 MDT 2009 x86_64 x86_64 x86_64 GNU/Linux

3) xm info attached.

4) brctl show
bridge name    bridge id        STP enabled    interfaces
eth0        8000.001517348ead    no        peth0

Note that at the moment I''m turning off iptables in dom0 to get domU
networking to work, I haven''t yet changed iptables to allow the
traffic. Not relevant to the problem, anyway.

Also note that I did some more digging on the panic mentioned in my first note,
which led me to [2] that has the same stack trace. This problem was related to
grub, so I goofed around with updating the grub in my domU disk, but it had no
effect. Since I''m providing the kernel and ramdisk, I assume the
domU''s grub isn''t involved much (if at all) during domU boot.

Thanks for any help you can provide,
Josh

[1]: http://blogs.sun.com/levon/entry/opensolaris_2009_06_guest_domain
[2]: https://opensolaris.org/jive/message.jspa?messageID=309582
-- 
This message posted from opensolaris.org
-----Inline Attachment Follows-----

_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org

Plus to first notice

Not sure your disk entries unmatching is not important :-

name = "storage"
uuid = "f4b21d4b-f5c7-46a4-8a62-7765fbdb2cb0"
vcpus = 1
memory = "1536"
kernel = "/var/xen/install/opensolaris/unix"
ramdisk = "/var/xen/install/opensolaris/x86.microroot"
extra = "/platform/i86xpv/kernel/amd64/unix - nowin -B
install_media=cdrom"
>  disk =
[''phy:/dev/loop0,xvdc:cdrom,r'',''phy:/dev/root_group/opensolaris_root_volume,xvda,w'']
vif = [''bridge=eth0'']
on_shutdown = "destroy"
on_reboot = "destroy"
on_crash = "destroy"

name = "storage"
uuid = "f4b21d4b-f5c7-46a4-8a62-7765fbdb2cb0"
vcpus = 1
memory = "1536"
# bootloader = ''/usr/bin/pygrub''
kernel = "/var/xen/boot/opensolaris/kernel"
ramdisk = "/var/xen/boot/opensolaris/ramdisk"
extra = "/platform/i86xpv/kernel/amd64/unix -B
zfs-bootfs=rpool/ROOT/opensolaris"
>  disk =
[''phy:/dev/root_group/opensolaris_root_volume,0,w'']
0 instead of xvda at install
vif = [''bridge=eth0'']

--- On Thu, 7/2/09, Josh Carter <no-reply@opensolaris.org> wrote:

From: Josh Carter <no-reply@opensolaris.org>
Subject: Re: [xen-discuss] can''t boot 2009.06 domU on Xen 3.4.1 /
CentOS 5.3 dom0
To: xen-discuss@opensolaris.org
Date: Thursday, July 2, 2009, 4:13 PM
> Please, post
> 1. your installation profile and shell commands to copy & solaris
kernel and microroot off the disk to Dom0.
> 2. uname -a
> 3. xm info
> 4. brctl show
Ok, here goes:

1) Install and run profiles attached. Shell commands to copy kernel/boot archive
are verbatim from [1].

2) uname -a (dom 0): Linux bashful.localdomain 2.6.18.8-xen #1 SMP Mon Jun 29
09:21:28 MDT 2009 x86_64 x86_64 x86_64 GNU/Linux

3) xm info attached.

4) brctl show
bridge name    bridge id        STP enabled    interfaces
eth0        8000.001517348ead    no        peth0

Note that at the moment I''m turning off iptables in dom0 to get domU
networking to work, I haven''t yet changed iptables to allow the
traffic. Not relevant to the problem, anyway.

Also note that I did some more digging on the panic mentioned in my first note,
which led me to [2] that has the same stack trace. This problem was related to
grub, so I goofed around with updating the grub in my domU disk, but it had no
effect. Since I''m providing the kernel and ramdisk, I assume the
domU''s grub isn''t involved much (if at all) during domU boot.

Thanks for any help you can provide,
Josh

[1]: http://blogs.sun.com/levon/entry/opensolaris_2009_06_guest_domain
[2]: https://opensolaris.org/jive/message.jspa?messageID=309582
-- 
This message posted from opensolaris.org
-----Inline Attachment Follows-----

_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org

The original blog was moved to:-
http://www.neuhalfen.name/2008/12/04/Installing_OpenSolaris_2008_11_in_a_Linux_dom0/

Quote:-

Getting ZFS boot ID

The new installer unmounts the new boot environment before we can ge a hold on
it. So we mount it by ourself:

$ beadm list
BE          Active Mountpoint Space Policy Created
--          ------ ---------- ----- ------ -------
opensolaris R      -          2.21G static 2008-12-05 06:58
$ pfexec beadm mount opensolaris /a

Update the boot archive

$ pfexec bootadm update-archive -R /a

And now get the kernel and boot_archive for you dom0 from
/a/platform/i86xpv/kernel/unix and /a/platform/i86pc/boot_archive.
You will need the ZFS-ID of your boot-environment. Your harddisk will be known
as c3d0, your boot-slice containing the ZPOOL is on c3d0s0. We need the ID of
the /ROOT/opensolaris zpool. First, determine the pool_guid:

$ zpool get guid rpool
NAME   PROPERTY  VALUE  SOURCE
rpool  guid      14025158204312238343  -

You could also have used zpool import like this — the choice is yours.

# pfexec zpool import
  pool: rpool
    id: 14025158204312238343
 state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
	the '-f' flag.
   see: http://www.sun.com/msg/ZFS-8000-EY
config:
	rpool       ONLINE
	c3d0s0    ONLINE

The guid by iself is useless – we need the ID of the ZPOOL inside the pool.

$ pfexec zdb -d -e 14025158204312238343
Dataset mos [META], ID 0, cr_txg 4, 512K, 84 objects
Dataset 14025158204312238343/ROOT/opensolaris@install [ZPL], ID 84, cr_txg 276,
2.21G, 101427 objects
Dataset 14025158204312238343/ROOT/opensolaris [ZPL], ID 50, cr_txg 72, 2.21G,
101427 objects
Dataset 14025158204312238343/ROOT [ZPL], ID 44, cr_txg 69, 18.0K, 4 objects
Dataset 14025158204312238343/dump [ZVOL], ID 36, cr_txg 13, 512M, 3 objects
Dataset 14025158204312238343/export/home/jens [ZPL], ID 68, cr_txg 80, 20.5K, 6
objects
Dataset 14025158204312238343/export/home [ZPL], ID 62, cr_txg 78, 19.0K, 5
objects
Dataset 14025158204312238343/export [ZPL], ID 56, cr_txg 76, 19.0K, 5 objects
Dataset 14025158204312238343/swap [ZVOL], ID 30, cr_txg 8, 16K, 3 objects
Dataset 14025158204312238343 [ZPL], ID 16, cr_txg 1, 72.0K, 12 objects

Your output might look different. Look out for /ROOT/opensolaris (here: 50).
Remember it!
-- 
This message posted from opensolaris.org
_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org

Replace your image storage , which is LVM
''phy:/dev/root_group/opensolaris_root_volume,xvda,w''
by ''phy :/dev/sda(X),xvda,w'' (just any  Linux fdisk partiton
type 83 ) and watch pygrub behavior after install. You might not need ZFS boot
ID any longer.
-- 
This message posted from opensolaris.org

Runtime snapshot at Xen 3.5 Unstable Dom0 (kernel 2.6.30 xenified via
Suse''s rebased patches)
-- 
This message posted from opensolaris.org

Susan,
> extra = "/platform/i86xpv/kernel/amd64/unix -B
zfs-bootfs=rpool/ROOT/opensolaris,bootpath=/xpvd/xdf@51712:a"
You''re a life saver. I had tried that before, but apparently with my
domU disk specified as "0" rather than "xvda". I went down
the path of determining my ZFS phys_path with zdb and did a lot of
head-scratching, but after double checking my disk specification and switching
it back to xvda, the domU booted up just fine. THANKS!

Best regards,
Josh
-- 
This message posted from opensolaris.org

Boris,
> Not sure your disk entries unmatching is not important :-
Turned out that was very important. At least with using an external kernel and
ramdisk, I needed to put the disk at xvda because otherwise
"bootpath=/xpvd/xdf@51712:a" didn''t make any sense to the
boot code.

Thanks and best regards,
Josh
-- 
This message posted from opensolaris.org

It should work via pygrub starting with Xen 3.4.0 Dom0 even with LVM image
stogare on
CentOS 5.3(2). View:-

http://foss-boss.blogspot.com/2009/06/opensolaris-200906-pv-under-xen-centos5.html

I don''t have have time to switch from Xen Host on top of Ubuntu 9.04 (
where LVM doesn''t work) to CentOS 5.3.

--- On Mon, 7/6/09, Josh Carter <no-reply@opensolaris.org> wrote:

From: Josh Carter <no-reply@opensolaris.org>
Subject: Re: [xen-discuss] can''t boot 2009.06 domU on Xen 3.4.1 /
CentOS 5.3 dom0
To: xen-discuss@opensolaris.org
Date: Monday, July 6, 2009, 3:53 PM

Boris,
> Not sure your disk entries unmatching is not important :-
Turned out that was very important. At least with using an external kernel and
ramdisk, I needed to put the disk at xvda because otherwise
"bootpath=/xpvd/xdf@51712:a" didn''t make any sense to the
boot code.

Thanks and best regards,
Josh
-- 
This message posted from opensolaris.org
_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org