zfs discuss - Jan 2009 - ZFS root mirror / moving disks to machine with different hostid

Hello,

My apologies if this has been discussed before or if this is
the wrong place to discuss Solaris 10 U6 issues..

I am investigating using ZFS as a possible replacement for SVM for
root disk mirroring.  So far, I have installed the system with the
new ZFS option in the text installer of U6.  I then attached my
second disk to the pool with a:

    zpool attach rpool c0t0d0s0 c0t1d0s0

Wait for resilvering, then install a ZFS bootblock on the new disk with:

    installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk
/dev/rdsk/c0t1d0s0

I can then boot off either disk, even when the other disk has been
removed.  Great.  However, if I place the disks into a different
machine and try to boot, I get:

    Executing last command: boot                                          
    Boot device: disk  File and args: 
    SunOS Release 5.10 Version Generic_137137-09 64-bit
    Copyright 1983-2008 Sun Microsystems, Inc.  All rights reserved.
    Use is subject to license terms.
    WARNING: pool ''rpool'' could not be loaded as it was last
accessed by another system (host:  hostid: 0x80c29c4c).  See:
http://www.sun.com/msg/ZFS-8000-EY
    NOTICE: 
    spa_import_rootpool: error 9
    
    Cannot mount root on /pci at 1f,0/pci at 1,1/scsi at 2/disk at 0,0:a fstype
zfs
    
    panic[cpu0]/thread=180e000: vfs_mountroot: cannot mount root
    
    000000000180b950 genunix:vfs_mountroot+358 (800, 200, 0, 1875c00, 189f800,
18c9000)
      %l0-3: 00000000010b9c00 00000000010b9ce0 000000000187baf8 00000000011e6800
      %l4-7: 00000000011e6800 00000000018cb800 0000000000000600 0000000000000200
    000000000180ba10 genunix:main+a0 (1815180, 180c000, 1839750, 18c5c00,
181b580, 1815000)
      %l0-3: 0000000001015400 0000000000000001 0000000070002000 0000000000000000
      %l4-7: 000000000183ec00 0000000000000000 000000000180c000 0000000000000000
    
    skipping system dump - no dump device configured
    rebooting...
    Resetting ... 

Is there a way to work through this?

Thanks,

Marcus

On January 30, 2009 2:26:36 PM -0800 Marcus Reid <marcus at
blazingdot.com>
wrote:
> I am investigating using ZFS as a possible replacement for SVM for
> root disk mirroring.
...
> Great.  However, if I place the disks into a different
> machine and try to boot, I get:
>
>     Executing last command: boot
>     Boot device: disk  File and args:
>     SunOS Release 5.10 Version Generic_137137-09 64-bit
>     Copyright 1983-2008 Sun Microsystems, Inc.  All rights reserved.
>     Use is subject to license terms.
>     WARNING: pool ''rpool'' could not be loaded as it was
last accessed by
> another system (host:  hostid: 0x80c29c4c)
...
> Is there a way to work through this?
I am doing the exact same thing.  I just updated to U6 (138889-03) and
am using zfs root mirror to replace SVM.  After learning the joys of
zfs root imagine my horror upon reading your message.

I just tested this, and luckily for me I am running x86, where the hostid
is software generated.  So it is identical when I boot from different
hardware, and zfs does not complain.

Here''s my first shot at workarounds.

1) clone the hostid.
2) keep the backup system in sync with the primary system as far as
installed software and configuration.  zones may make this easier for
you.  i used to keep zones in sync on different machines for failover
purposed, it''s not too awful.

neither of those might be useful if you have one backup system for
multiple primary systems.

But what is probably best,

3) when it comes time to make your backup system act as the failed system,
first boot it from the network or from cdrom, or possibly you could have
it already running, ready to go.  forcibly mount the root pool (just
as any other pool) which will record the hostid, then reboot from that
drive.

A kernel option to forcibly import the root pool would make things a
lot easier.  Looking at kernel(1M) and boot(1M) there doesn''t appear
to be one.  Maybe OpenSolaris has it?

-frank

On Mon, Feb 02, 2009 at 08:41:13PM -0800, Frank Cusack
wrote:
> On January 30, 2009 2:26:36 PM -0800 Marcus Reid <marcus at
blazingdot.com>
> wrote:
>
> But what is probably best,
> 
> 3) when it comes time to make your backup system act as the failed system,
> first boot it from the network or from cdrom, or possibly you could have
> it already running, ready to go.  forcibly mount the root pool (just
> as any other pool) which will record the hostid, then reboot from that
> drive.
Hi Frank,

Thanks for the response.  It turns out that there is a way to do this
after the fact.  Doing a ''boot disk0 -F failsafe'' at the ok
prompt
(replace disk0 with another device if needed).  This gets you to a
failsafe shell, where a ''zpool import -f rpool'' will import
the pool
even if it was previously imported by another host.  Once that''s done
the system comes up normally.

I wasn''t aware of the failsafe shell before, and it seems like a real
lifesaver.  Hat tip to Scott Dickson at Sun for the tip.  Also a
(re) read of the boot manpage was long overdue for me, as there
seems to be a lot of new stuff.

Marcus

On Mon, Feb 02, 2009 at 08:41:13PM -0800, Frank Cusack
wrote:
> On January 30, 2009 2:26:36 PM -0800 Marcus Reid <marcus at
blazingdot.com>
> wrote:
> >I am investigating using ZFS as a possible replacement for SVM for
> >root disk mirroring.
> 
> I am doing the exact same thing.  I just updated to U6 (138889-03) and
> am using zfs root mirror to replace SVM.  After learning the joys of
> zfs root imagine my horror upon reading your message.
I thought I''d add a couple more observations while we''re on
the topic:

  - if a zpool has not been imported yet, such as while a
    system is booting, and you have replaced a disk with a
    different one, zfs gets confused and thinks that the  
    mirror is corrupted and will refuse to import it at all.
    So, make sure that you have a running machine with an   
    imported root zpool before you install your replacement.
                                                            
    This appears to only be a problem for a replacement disk
    that has a valid label on it.  If you dd zeroes over the
    label and reboot, you can import the pool with the
    mirror in it (degraded of course).

I was able to do something insane and not recommended last
night, upgrading an old remote machine over the network by
breaking its old mirror, writing over a disk with a dd image
of c0t0d0s2, rebooting on the new disk, and using failsafe
mode to fix the disk (by doing the above mentioned zeroing
the label of the other old disk before rebooting again and
then importing.)

So far I haven''t run into any badness that I couldn''t fix
with ZFS root, so I''m pretty enthused about it.  This is
all SPARC here.

Marcus