zfs discuss - Dec 2009 - ZFS Boot Recovery after Motherboard Death

After a Power Outage last week my server wouldn''t turn on anymore (yes,
UPS ...).  I tracked it down to a motherboard failure and ordered a new MB, CPU
& Memory ...

After swapping it all out the system failed to boot with:


NOTICE: error reading device label
NOTICE:
  ***************************************************
  *  This device is not bootable!                   *
  *  It is either offlined or detached or faulted.  *
  *  Please try to boot from a different device.    *
  ***************************************************
NOTICE: spa_import_rootpool: error 19
Cannot mount root on /pci at 0,0/pci1458,b003 at 8/disk at 0,0:a fstype zfs

panic[cpu0]/thread=fffffffffbc2cfe0: vfs_mountroot: cannot mount root


After digging around in the dark for a while, I downloaded a LiveCD and looked
at my zpool from a running OS.  Doh!

The host identity had - of course - changed with the new motherboard and it no
longer recognised the zpool as its own.  ''zpool import -f
rpool'' to take ownership, reboot and it all worked no problem (which
was amazing in itself as I had switched from AMD to Intel ...).

Any chance this bit of info could be added to
http://www.solarisinternals.com/wiki/index.php/ZFS_Troubleshooting_Guide (to
save someone else some time when trying to recover a host into the wee hours of
the morning!)?

On the other hand the NOTICE could also be improved upon to say that the pool
cannot be imported because it is owned by someone else (it was not offline,
detached or faulted). Should I raise this as a bug?

thanks! - mo

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2196 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091212/8fae7ec6/attachment.bin>

On Sat, 2009-12-12 at 00:22 +0000, Moritz Willers wrote:
> The host identity had - of course - changed with the new motherboard
>  and it no longer recognised the zpool as its own.  ''zpool import
-f
>  rpool'' to take ownership, reboot and it all worked no problem
(which
>  was amazing in itself as I had switched from AMD to Intel ...).
Do I understand correctly if I read this as: OpenSolaris is able to
switch between systems without reinstalling? Just a zfs import -f and
everything runs? Wow, that would be an improvemment and would make
things more like *BSD/linux.

On Dec 12, 2009, at 12:53 AM, dick hoogendijk wrote:
> On Sat, 2009-12-12 at 00:22 +0000, Moritz Willers wrote:
>
>> The host identity had - of course - changed with the new motherboard
>> and it no longer recognised the zpool as its own.  ''zpool
import -f
>> rpool'' to take ownership, reboot and it all worked no problem
(which
>> was amazing in itself as I had switched from AMD to Intel ...).
>
> Do I understand correctly if I read this as: OpenSolaris is able to
> switch between systems without reinstalling? Just a zfs import -f and
> everything runs? Wow, that would be an improvemment and would make
> things more like *BSD/linux.
Solaris has been able to do that for 20+ years.  Why do you think
it should be broken now?
  -- richard

On Sat, 2009-12-12 at 09:08 -0800, Richard Elling wrote:
> On Dec 12, 2009, at 12:53 AM, dick hoogendijk wrote:
> 
> > On Sat, 2009-12-12 at 00:22 +0000, Moritz Willers wrote:
> >
> >> The host identity had - of course - changed with the new
motherboard
> >> and it no longer recognised the zpool as its own.  ''zpool
import -f
> >> rpool'' to take ownership, reboot and it all worked no
problem (which
> >> was amazing in itself as I had switched from AMD to Intel ...).
> >
> > Do I understand correctly if I read this as: OpenSolaris is able to
> > switch between systems without reinstalling? Just a zfs import -f and
> > everything runs? Wow, that would be an improvemment and would make
> > things more like *BSD/linux.
> 
> Solaris has been able to do that for 20+ years.  Why do you think
> it should be broken now?
Because, like I said, I always understood it was very difficult to
change disks to another system and run the installed solaris version on
that new hardware.

-- 
Dick Hoogendijk -- PGP/GnuPG key: 01D2433D
+ http://nagual.nl/ | SunOS 10u7 5/09 | OpenSolaris 2010.02 b129
+ All that''s really worth doing is what we do for others (Lewis Carrol)

On Sat, 12 Dec 2009, dick hoogendijk wrote:
>
> Because, like I said, I always understood it was very difficult to
> change disks to another system and run the installed solaris version on
> that new hardware.
A place where I used to work had several thousand Sun workstations and 
I noticed that if a system drive failed, the system administrator 
would just walk up with a replacement drive that had Solaris 
pre-installed, do the swap, and the system was running in a few 
minutes.  Of course that was quite a while ago (when the world was a 
cooler place) and things could have become broken since then.

Bob
--
Bob Friesenhahn
bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,    http://www.GraphicsMagick.org/

On Sat, Dec 12, 2009 at 18:08, Richard Elling <richard.elling at
gmail.com> wrote:
> On Dec 12, 2009, at 12:53 AM, dick hoogendijk wrote:
>
>> On Sat, 2009-12-12 at 00:22 +0000, Moritz Willers wrote:
>>
>>> The host identity had - of course - changed with the new
motherboard
>>> and it no longer recognised the zpool as its own. ?''zpool
import -f
>>> rpool'' to take ownership, reboot and it all worked no
problem (which
>>> was amazing in itself as I had switched from AMD to Intel ...).
>>
>> Do I understand correctly if I read this as: OpenSolaris is able to
>> switch between systems without reinstalling? Just a zfs import -f and
>> everything runs? Wow, that would be an improvemment and would make
>> things more like *BSD/linux.
>
> Solaris has been able to do that for 20+ years. ?Why do you think
> it should be broken now?
Solaris has _not_ been able to do that for 20+ years. In fact Sun has
always recommended a reinstall. You could do it if you really knew
how, but it was not easy.

If you switch between identical system it will of course work fine
(before zfs that is, now you may have to import the pool on the new
system).

On 12-Dec-09, at 1:32 PM, Mattias Pantzare wrote:
> On Sat, Dec 12, 2009 at 18:08, Richard Elling  
> <richard.elling at gmail.com> wrote:
>> On Dec 12, 2009, at 12:53 AM, dick hoogendijk wrote:
>>
>>> On Sat, 2009-12-12 at 00:22 +0000, Moritz Willers wrote:
>>>
>>>> The host identity had - of course - changed with the new  
>>>> motherboard
>>>> and it no longer recognised the zpool as its own. 
''zpool import -f
>>>> rpool'' to take ownership, reboot and it all worked no
problem
>>>> (which
>>>> was amazing in itself as I had switched from AMD to Intel ...).
>>>
>>> Do I understand correctly if I read this as: OpenSolaris is able to
>>> switch between systems without reinstalling? Just a zfs import -f  
>>> and
>>> everything runs? Wow, that would be an improvemment and would make
>>> things more like *BSD/linux.
>>
>> Solaris has been able to do that for 20+ years.  Why do you think
>> it should be broken now?
>
> Solaris has _not_ been able to do that for 20+ years. In fact Sun has
> always recommended a reinstall. You could do it if you really knew
> how, but it was not easy.
>
> If you switch between identical system it will of course work fine

Linux can''t do it either, of course, unless one is deliberately using  
a sufficiently generic kernel.

--Toby
(who doesn''t really wish to start an O/S pissing contest)

> (before zfs that is, now you may have to import the pool on the new
> system).
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

On Dec 12, 2009, at 10:32 AM, Mattias Pantzare wrote:
> On Sat, Dec 12, 2009 at 18:08, Richard Elling <richard.elling at
gmail.com
> > wrote:
>> On Dec 12, 2009, at 12:53 AM, dick hoogendijk wrote:
>>
>>> On Sat, 2009-12-12 at 00:22 +0000, Moritz Willers wrote:
>>>
>>>> The host identity had - of course - changed with the new  
>>>> motherboard
>>>> and it no longer recognised the zpool as its own. 
''zpool import -f
>>>> rpool'' to take ownership, reboot and it all worked no
problem
>>>> (which
>>>> was amazing in itself as I had switched from AMD to Intel ...).
>>>
>>> Do I understand correctly if I read this as: OpenSolaris is able to
>>> switch between systems without reinstalling? Just a zfs import -f  
>>> and
>>> everything runs? Wow, that would be an improvemment and would make
>>> things more like *BSD/linux.
>>
>> Solaris has been able to do that for 20+ years.  Why do you think
>> it should be broken now?
>
> Solaris has _not_ been able to do that for 20+ years. In fact Sun has
> always recommended a reinstall. You could do it if you really knew
> how, but it was not easy.
A flash archive is merely  a cpio image of an existing system wrapped
by clever scripts that edit /etc/vfstab and reset the sysidcfg.  You can
do this by hand or script quite easily and many of us have done so since
the late 1980s. With ZFS it is a little bit easier, because you no  
longer
have to edit /etc/vfstab.

As to what is "supported," I know of nobody at Sun that has a list of
what is or what is not "supported." Clearly, it is easier to CYA by
falling back to "reinstallation required."
> If you switch between identical system it will of course work fine
> (before zfs that is, now you may have to import the pool on the new
> system).
When I was designing appliances at Sun, I kept having to fight with
marketing because we developed the software stack to work across
all platforms of the same architecture. Marketing was convinced that
you could not create a software stack that worked the same on a lowly
desktop as a F15K. Go figure.
  -- richard

It may not be supported, but you can swap drives between systems and it 
does work very well.

I did a Solaris 8 -> Solaris 10 migration on ~200 systems in 2007.  I 
had a set of systems that I jumpstarted, then, once the system was 
built, I pulled the drives and placed them in the new system.

It worked very well and I had minimal downtime for people using these 
servers.

Jerry


On 12/12/09 12:32, Mattias Pantzare wrote:
> On Sat, Dec 12, 2009 at 18:08, Richard Elling <richard.elling at
gmail.com> wrote:
>> On Dec 12, 2009, at 12:53 AM, dick hoogendijk wrote:
>>
>>> On Sat, 2009-12-12 at 00:22 +0000, Moritz Willers wrote:
>>>
>>>> The host identity had - of course - changed with the new
motherboard
>>>> and it no longer recognised the zpool as its own. 
''zpool import -f
>>>> rpool'' to take ownership, reboot and it all worked no
problem (which
>>>> was amazing in itself as I had switched from AMD to Intel ...).
>>> Do I understand correctly if I read this as: OpenSolaris is able to
>>> switch between systems without reinstalling? Just a zfs import -f
and
>>> everything runs? Wow, that would be an improvemment and would make
>>> things more like *BSD/linux.
>> Solaris has been able to do that for 20+ years.  Why do you think
>> it should be broken now?
> 
> Solaris has _not_ been able to do that for 20+ years. In fact Sun has
> always recommended a reinstall. You could do it if you really knew
> how, but it was not easy.
> 
> If you switch between identical system it will of course work fine
> (before zfs that is, now you may have to import the pool on the new
> system).

On Sat, Dec 12, 2009 at 5:08 PM, Richard Elling
<richard.elling at gmail.com> wrote:
> On Dec 12, 2009, at 12:53 AM, dick hoogendijk wrote:
>
>> Do I understand correctly if I read this as: OpenSolaris is able to
>> switch between systems without reinstalling? Just a zfs import -f and
>> everything runs? Wow, that would be an improvemment and would make
>> things more like *BSD/linux.
>
> Solaris has been able to do that for 20+ years. ?Why do you think
> it should be broken now?
Only for "sufficiently similar" systems. There''s historically
been an
awful lot of detailed system configuration baked into an installed
image. Yes, you can get rid of it, but the idea that you could pull
drives from a failed system and put them into any old system they
might happen to fit in and expect it to just work has always been
optimistic. The advantage of zfs is that it abstracts a lot of that away.

-- 
-Peter Tribble
http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/

On Dec 13, 2009, at 12:17 PM, Peter Tribble wrote:
> On Sat, Dec 12, 2009 at 5:08 PM, Richard Elling
> <richard.elling at gmail.com> wrote:
>> On Dec 12, 2009, at 12:53 AM, dick hoogendijk wrote:
>>
>>> Do I understand correctly if I read this as: OpenSolaris is able to
>>> switch between systems without reinstalling? Just a zfs import -f  
>>> and
>>> everything runs? Wow, that would be an improvemment and would make
>>> things more like *BSD/linux.
>>
>> Solaris has been able to do that for 20+ years.  Why do you think
>> it should be broken now?
>
> Only for "sufficiently similar" systems. There''s
historically been an
> awful lot of detailed system configuration baked into an installed
> image. Yes, you can get rid of it, but the idea that you could pull
> drives from a failed system and put them into any old system they
> might happen to fit in and expect it to just work has always been
> optimistic. The advantage of zfs is that it abstracts a lot of that  
> away.
I''ll disagree and have successful projects that prove it.  However,
this is way off topic.  I think we can all agree that forgetting how to
type "vi /etc/vfstab" is a win-win for everyone :-)
  -- richard