zfs discuss - Sep 2008 - zpool import of bootable root pool renders it unbootable

I am running OpenSolaris 2008.05 as a PV guest under Xen.  If you import the
bootable root pool of a VM into another Solaris VM, the root pool is no longer
bootable.

It is related to the device associated to the pool, which is originally c4d0s0,
but on import (-f) becomes c0d2s0 in this case.  Afterwards, booting the
original image results in a kernel panic because, I think, zfs_mountroot()
cannot mount the root path (which is evidently now wrong).

I this fixable?  How does one "mount" (import) a bootable zpool
without wrecking it?  This is something that is commonly done under
virtualization platforms, e.g., to manage the contents of a VM from another VM,
or to perform a file-system level copy of the contents of a VM to another
device.

Any insight would be appreciated.
--
This message posted from opensolaris.org

Stephen Quintero <stephen at 3tera.com> writes:
> I am running OpenSolaris 2008.05 as a PV guest under Xen. If you
> import the bootable root pool of a VM into another Solaris VM, the
> root pool is no longer bootable.
I had a similar problem: After installing and booting Opensolaris
2008.05, I succeded to lock myself out through some passwd/shadow
inconsistency (totally my own fault). Not a problem, I thought -- I
booted from the install disk, imported the root pool, fixed the
inconsistency, and rebooted. Lo, instant panic.

No idea why, though, I am not that familiar with the underlying
code. I just did a reinstall.

Regards, Juergen.

Hello Juergen,

Tuesday, September 30, 2008, 5:43:56 PM, you wrote:

JN> Stephen Quintero <stephen at 3tera.com> writes:
>> I am running OpenSolaris 2008.05 as a PV guest under Xen. If you
>> import the bootable root pool of a VM into another Solaris VM, the
>> root pool is no longer bootable.
JN> I had a similar problem: After installing and booting Opensolaris
JN> 2008.05, I succeded to lock myself out through some passwd/shadow
JN> inconsistency (totally my own fault). Not a problem, I thought -- I
JN> booted from the install disk, imported the root pool, fixed the
JN> inconsistency, and rebooted. Lo, instant panic.

JN> No idea why, though, I am not that familiar with the underlying
JN> code. I just did a reinstall.

I hit the same issue - once I tried to boot OS from within virtualbox
with disk partition exposed to VB - kernel couldn''t mount root fs
either from VB or directly from notebook - I had to import/export pool
while booting from CD. I haven''t investigated it further but
I''m
surprised it''s not working OOB.


-- 
Best regards,
 Robert                            mailto:milek at task.gda.pl
                                       http://milek.blogspot.com

On Tue, 30 Sep 2008, Robert Milkowski wrote:
> Hello Juergen,
>
> Tuesday, September 30, 2008, 5:43:56 PM, you wrote:
>
> JN> Stephen Quintero <stephen at 3tera.com> writes:
>
>>> I am running OpenSolaris 2008.05 as a PV guest under Xen. If you
>>> import the bootable root pool of a VM into another Solaris VM, the
>>> root pool is no longer bootable.
>
> JN> I had a similar problem: After installing and booting Opensolaris
> JN> 2008.05, I succeded to lock myself out through some passwd/shadow
> JN> inconsistency (totally my own fault). Not a problem, I thought -- I
> JN> booted from the install disk, imported the root pool, fixed the
> JN> inconsistency, and rebooted. Lo, instant panic.
>
> JN> No idea why, though, I am not that familiar with the underlying
> JN> code. I just did a reinstall.
>
> I hit the same issue - once I tried to boot OS from within virtualbox
> with disk partition exposed to VB - kernel couldn''t mount root fs
> either from VB or directly from notebook - I had to import/export pool
> while booting from CD. I haven''t investigated it further but
I''m
> surprised it''s not working OOB.
I think this is 6737463

-- Dave

I have not tried importing bootable root pools onto other VMs, but there have
been recent ZFS bug fixes in the area of importing and exporting bootable root
pools - the panic might not occur on Solaris Nevada releases after approximately
97.

There are still issues with renaming of bootable root pools - particularly if
they are renamed during import - "newpool" from zpool(1m).  If you
import with a different name, at the moment, you will have to export, then
import by the original name before it can be booted without GRUB menu changes.

Check your Solaris version, check to see if your zpool import is using an
alternate pool name. If so, try re-importing using the original name before
trying to reboot.  Let us know what happens.
--
This message posted from opensolaris.org

Hello David,

Tuesday, September 30, 2008, 10:57:45 PM, you wrote:

DF> On Tue, 30 Sep 2008, Robert Milkowski wrote:
>> Hello Juergen,
>>
>> Tuesday, September 30, 2008, 5:43:56 PM, you wrote:
>>
>> JN> Stephen Quintero <stephen at 3tera.com> writes:
>>
>>>> I am running OpenSolaris 2008.05 as a PV guest under Xen. If
you
>>>> import the bootable root pool of a VM into another Solaris VM,
the
>>>> root pool is no longer bootable.
>>
>> JN> I had a similar problem: After installing and booting
Opensolaris
>> JN> 2008.05, I succeded to lock myself out through some
passwd/shadow
>> JN> inconsistency (totally my own fault). Not a problem, I thought
-- I
>> JN> booted from the install disk, imported the root pool, fixed the
>> JN> inconsistency, and rebooted. Lo, instant panic.
>>
>> JN> No idea why, though, I am not that familiar with the underlying
>> JN> code. I just did a reinstall.
>>
>> I hit the same issue - once I tried to boot OS from within virtualbox
>> with disk partition exposed to VB - kernel couldn''t mount root
fs
>> either from VB or directly from notebook - I had to import/export pool
>> while booting from CD. I haven''t investigated it further but
I''m
>> surprised it''s not working OOB.
DF> I think this is 6737463

By quickly looking at stack trace IIIRC mine was different - actually
it didn''t panicked it couldn''t mount rootfs and rebooted.

Basically what I did - I has a working OS (b96 IIRC) along with
Windows on my notebook. Both systems were booting properly.
I installed VB on Windows and created a VM with a Solaris partition
exposed as a raw device - VB showed me Grub I chose OS and it failed
on mounting rootfs.
After Windows reboot I couldn''t boot into OS (from HW not VB).
I booted from LiveCD, imported the pool, exported, rebooted again and
now it booted fine from bare metal - haven''t tried from VB again.



-- 
Best regards,
 Robert                            mailto:milek at task.gda.pl
                                       http://milek.blogspot.com

I came across this bug in a similar way myself. The explanation given by Stephen
Hahn is this:

----------
For a while, the boot-archive on 2008.nn systems included a copy of
 zpool.cache.  Recent versions do not make this mistake.  Delete and
 regenerate your boot archive, and you should be able to make the
 transfer.  See

 http://mail.opensolaris.org/pipermail/indiana-discuss/2008-August/008341.html

 and following.
-----------

If you install a system from scratch with the latest test build of OpenSolaris
2008.11 (which you can get from genunix.org) then you won''t have this
problem. Solaris Express Community Edition (SXCE) is also not affected by this
bug.

Cheers

Andrew.
--
This message posted from opensolaris.org

David Finberg <David.Finberg at Sun.COM> writes:
>> JN> I had a similar problem: After installing and booting
Opensolaris
>> JN> 2008.05, I succeded to lock myself out through some
passwd/shadow
>> JN> inconsistency (totally my own fault). Not a problem, I thought
-- I
>> JN> booted from the install disk, imported the root pool, fixed the
>> JN> inconsistency, and rebooted. Lo, instant panic.
[...]
> I think this is 6737463
Quite possible. Only the the machine rebooted so fast that I barely
could see the word "panic", let alone anything more specific. :-}

In a development version of a system I am not *that* suprised by a
panic like this; I was rather baffled about the system not being
able, apparently, to boot from the (just so slightly, I thought)
externally modified root pool.

Regards, Juergen.

Hello andrew,

Thursday, October 2, 2008, 7:56:38 PM, you wrote:

a> I came across this bug in a similar way myself. The explanation given by
Stephen Hahn is this:

a> ----------
a> For a while, the boot-archive on 2008.nn systems included a copy of
a>  zpool.cache.  Recent versions do not make this mistake.  Delete and
a>  regenerate your boot archive, and you should be able to make the
a>  transfer.  See

a> 
a>
http://mail.opensolaris.org/pipermail/indiana-discuss/2008-August/008341.html

a>  and following.
a> -----------

a> If you install a system from scratch with the latest test build of
a> OpenSolaris 2008.11 (which you can get from genunix.org) then you
a> won''t have this problem. Solaris Express Community Edition (SXCE)
a> is also not affected by this bug.

a> Cheers

a> Andrew.


I''ve upgraded to b98, checked if zpool.cache is not being added to
boot archive and tried to boot from VB by presenting a prtition to it.
It didn''t.

I got


-- 
Best regards,
 Robert                            mailto:milek at task.gda.pl
                                       http://milek.blogspot.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 11.JPG
Type: image/jpeg
Size: 59044 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081006/c25f6906/attachment.jpe>

> I''ve upgraded to b98, checked if zpool.cache is not
> being added to
> boot archive and tried to boot from VB by presenting
> a prtition to it.
> It didn''t.
I got it working by installing a new build of OpenSolaris 2008.11 from scratch
rather than upgrading, but deleting zpool.cache, deleting both boot archives,
then doing a "bootadm update-archive" should work.

Cheers

Andrew.
--
This message posted from opensolaris.org

> Cannot mount root on /pci at 0,0/pci103c,30aa at 1f,2/disk at 0,0:a fstype
zfs
Is that physical device path correct for your new  system?

Or is this the physical device path (stored on-disk in the zpool label)
from some other system?   In this case you may be able to work around
the problem by passing a "-B bootpath=..."  option to the kernel

e.g. something like this:

kernel$ /platform/i86pc/kernel/$ISADIR/unix -B $ZFS-BOOTFS,bootpath="/pci
at 0,0/pci-ide at 1,1/ide at 0/cmdk at 0,0:a"


You can find out the correct physical device path string
for the zfs root disk by booting the system from the optical 
installation media, and running the format utility.

OTOH, if you already have booted from the optical installation
media, it''s easiest to just import the root zpool from the
installation system, because that''ll update the physical device
path in the zpool''s label on disk (and it clears the hostid stored in
the zpool label - another problem that could prevent mounting
the zfs root).
--
This message posted from opensolaris.org

Thanks for all of your input.  I intalled 2008.11 build 98 as an HVM guest under
Xen:
 
If you make a block-level copy of the boot pool and attach it as a disk on the
original VM, "zpool import" does not recognize it.  If you attach a
non-root pool as a disk, "zpool import" does recognize it.  So, I am
unable to test whether build 98 has fixed the problem.
--
This message posted from opensolaris.org

Hello J?rgen,

Monday, October 6, 2008, 6:27:54 PM, you wrote:
>> Cannot mount root on /pci at 0,0/pci103c,30aa at 1f,2/disk at 0,0:a
fstype zfs
JK> Is that physical device path correct for your new  system?

JK> Or is this the physical device path (stored on-disk in the zpool label)
JK> from some other system?   In this case you may be able to work around
JK> the problem by passing a "-B bootpath=..."  option to the
kernel

JK> e.g. something like this:

JK> kernel$ /platform/i86pc/kernel/$ISADIR/unix -B
JK> $ZFS-BOOTFS,bootpath="/pci at 0,0/pci-ide at 1,1/ide at 0/cmdk at
0,0:a"


JK> You can find out the correct physical device path string
JK> for the zfs root disk by booting the system from the optical 
JK> installation media, and running the format utility.

JK> OTOH, if you already have booted from the optical installation
JK> media, it''s easiest to just import the root zpool from the
JK> installation system, because that''ll update the physical device
JK> path in the zpool''s label on disk (and it clears the hostid
stored in
JK> the zpool label - another problem that could prevent mounting
JK> the zfs root).


Again, what I''m trying to do is to boot the same OS from physical
drive - once natively on my notebook, the other time from withing
Virtualbox. There are two problems, at least. First is the bootpath as
in VB it emulates the disk as IDE while booting natively it is sata.
The other one seems to be hostid stored in a pool.

Yes, importing the pool when booted from cdrom is a working
workaround... but it''s not practical.

The other thing is - if one boots from another media with root-fs on
zfs and tries to import other root-fs on zfs, there will be the
problem with both pools having the same name. I could import the pool
under different name but one can''t export the pool while reverting it
back to original name. So far I workaround it by booting from media
where root-fs is not on zfs - probably an ability to import same pool
name or export with name change at the same time would be useful for
some cases.


-- 
Best regards,
 Robert Milkowski                            mailto:milek at task.gda.pl
                                       http://milek.blogspot.com

> Again, what I''m trying to do is to boot the same OS from physical
> drive - once natively on my notebook, the other time from withing
> Virtualbox. There are two problems, at least. First is the bootpath as
> in VB it emulates the disk as IDE while booting natively it is sata.
When I started experimenting with installing SXCE to an USB flash
memory stick, which should be bootable on different machines,
I initially worked around this problem by creating multiple
/boot/grub/menu.lst boot entries, one for each supported 
bootable machine.  The difference between the grub boot entries
was the "-B bootpath=/physical/device/path" option.


Fortunately, this has become much easier in recent builds,
because zfs boot is now able to open the pool by using a
disk unique "devid" in addition to using physical device paths.
Whatever the physical device path will be on a random selected
x86 machine where I try to boot my usb flash memory stick,
the sd driver will always generate the same unique "devid" for
the flash memory stick, and zfs boot is able to find and open
the correct usb storage device in the system that has the
desired "devid" for the pool.


In case sata on the notebook will create the same "devid" for the
disk as virtualbox with p-ata, the zpool should be bootable just fine
on the two different boxes.  But apparently the "devid" created for
the disk with sata on the notebook is different from the "devid"
created for the disk when running under virtualbox...
(that is, pool open by physical device path and by devid fails)


I guess what we need is the fix for this bug, which allows to open
the pool by the boot disk''s unique "guid":

Bug ID   	 6513775
Synopsis 	zfs root disk portability
http://bugs.opensolaris.org/view_bug.do?bug_id=6513775
> The other one seems to be hostid stored in a pool.
This shouldn''t be a problem for x86, because the
hostid is stored in a file (sysinit kernel module) in the
root filesystem, on x86.   Wherever you boot that disk,
the hostid will move with it.

Well, unless you boot some other installed Solaris /
OpenSolaris system (has a unique hostid / sysinit file)
and import that zfs root pool.  In this case the hostid
stored in the zpool label will change.

This should change in build 100, with the putback for 
this bug:

Bug ID   	 6716241
Synopsis 	Changing hostid, by moving in a new sysinit file, panics a zfs root
file system
http://bugs.opensolaris.org/view_bug.do?bug_id=6716241

AFAIR, a hostid mismatch will be ignored when mounting a zfs
root file system.
--
This message posted from opensolaris.org