zfs discuss - Jul 2008 - Cannot attach mirror to SPARC zfs root pool

I just wanted to attach a second mirror to a ZFS root pool on an Ultra
1/170E running snv_93.

I''ve followed the workarounds for CR 6680633 and 6680633 from the ZFS
Admin
Guide, but booting from the newly attached mirror fails like so:

Boot device: disk  File and args:                                     

Can''t mount root
Fast Data Access MMU Miss

while the original side of the mirror works just fine.

Any advice on what could be wrong here?

	Rainer

-----------------------------------------------------------------------------
Rainer Orth, Faculty of Technology, Bielefeld University

On Tue, 22 Jul 2008, Rainer Orth wrote:
> I just wanted to attach a second mirror to a ZFS root pool on an Ultra 
> 1/170E running snv_93.
>
> I''ve followed the workarounds for CR 6680633 and 6680633 from the
ZFS
> Admin Guide, but booting from the newly attached mirror fails like so:
I think you''re running into CR 6668666.  I''d try manually
running
instlalboot on the new disk and see if that fixes it.


Regards,
markm

Mark J Musante writes:
> On Tue, 22 Jul 2008, Rainer Orth wrote:
> 
> > I just wanted to attach a second mirror to a ZFS root pool on an Ultra
> > 1/170E running snv_93.
> >
> > I''ve followed the workarounds for CR 6680633 and 6680633 from
the ZFS
> > Admin Guide, but booting from the newly attached mirror fails like so:
> 
> I think you''re running into CR 6668666.  I''d try manually
running
oops, cut-and-paste error on my part: 6668666 was one of the two CRs
mentioned in the zfs admin guide which I worked around.
> instlalboot on the new disk and see if that fixes it.
Unfortunately, it didn''t.  Reconsidering now, I see that I ran
installboot
against slice 0 (reduced by 1 sector as required by CR 6680633) instead of
slice 2 (whole disk).  Doing so doesn''t fix the problem either, though.

Regards.
	Rainer

-----------------------------------------------------------------------------
Rainer Orth, Faculty of Technology, Bielefeld University

Rainer Orth <ro at techfak.uni-bielefeld.de> writes:
> > instlalboot on the new disk and see if that fixes it.
> 
> Unfortunately, it didn''t.  Reconsidering now, I see that I ran
installboot
> against slice 0 (reduced by 1 sector as required by CR 6680633) instead of
> slice 2 (whole disk).  Doing so doesn''t fix the problem either,
though.
I''ve found out what the problem was: I didn''t specify the -F
zfs option to
installboot, so only half of the ZFS bootblock was written.  This is a
combination of two documentation bugs and a terrible interface:

* With the introduction of zfs boot, installboot got a new -F <fstype>
  option.  Unfortunately, this is documented neither on installboot(1M)
  (which wasn''t update at all, it seems) nor in the ZFS Admin Guide
  (p.80, workaround for CR 6668666).

* Apart from that, I''ve never understood why it is necessary to specify
the
  full path to the bootblock to installboot like this

installboot /usr/platform/`uname -i`/lib/fs/<fstype>/bootblk
/dev/rdsk/c0t0d0s0

  It would be far easier to just specify the fstype (or even let
  installboot figure that out by itself using fstyp) than having to give
  the full pathname.  In that case, installboot could just dd the whole
  bootblk file instead of hardcoding the block counts for the different
  filesystem types (probably to avoid corrupting the filesystem if the user
  gives a file that is not a bootblock).

Overall, a terrible mess ;-(

Regards.
	Rainer

-- 
-----------------------------------------------------------------------------
Rainer Orth, Faculty of Technology, Bielefeld University

Rainer Orth wrote:
> Rainer Orth <ro at techfak.uni-bielefeld.de> writes:
>
>   
>>> instlalboot on the new disk and see if that fixes it.
>>>       
>> Unfortunately, it didn''t.  Reconsidering now, I see that I ran
installboot
>> against slice 0 (reduced by 1 sector as required by CR 6680633) instead
of
>> slice 2 (whole disk).  Doing so doesn''t fix the problem
either, though.
>>     
>
> I''ve found out what the problem was: I didn''t specify the
-F zfs option to
> installboot, so only half of the ZFS bootblock was written.  This is a
> combination of two documentation bugs and a terrible interface:
>   
Mainly because there is no -F option?
> * With the introduction of zfs boot, installboot got a new -F
<fstype>
>   option.  Unfortunately, this is documented neither on installboot(1M)
>   (which wasn''t update at all, it seems) nor in the ZFS Admin
Guide
>   (p.80, workaround for CR 6668666).
>
> * Apart from that, I''ve never understood why it is necessary to
specify the
>   full path to the bootblock to installboot like this
>
> installboot /usr/platform/`uname -i`/lib/fs/<fstype>/bootblk
/dev/rdsk/c0t0d0s0
>   
That is because installboot is simply a wrapper for dd.
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/psm/stand/bootblks/ufs/i386/installboot.sh

The first argument is copied to the second argument using dd.
>   It would be far easier to just specify the fstype (or even let
>   installboot figure that out by itself using fstyp) than having to give
>   the full pathname.  In that case, installboot could just dd the whole
>   bootblk file instead of hardcoding the block counts for the different
>   filesystem types (probably to avoid corrupting the filesystem if the user
>   gives a file that is not a bootblock).
>
> Overall, a terrible mess ;-(
>
>   
I think that it should be very unusual that installboot would be run
interactively.  That is really no excuse for making it only slightly
smarter than dd, but it might be hard to justify changes unless some
kind person were to submit a bug with an improved implementation
(would make a good short project for someone :-)
 -- richard

Richard Elling writes:
> > I''ve found out what the problem was: I didn''t
specify the -F zfs option to
> > installboot, so only half of the ZFS bootblock was written.  This is a
> > combination of two documentation bugs and a terrible interface:
> >   
> 
> Mainly because there is no -F option?
Huh?  From /usr/sbin/installboot:

COUNT=15

while getopts F: a; do
        case $a in
        F) case $OPTARG in
           ufs) COUNT=15;;
           hsfs) COUNT=15;;
           zfs) COUNT=31;;
           *) away 1 "$OPTARG: Unknown fstype";;
           esac;;

Without -F zfs, only part of the zfs bootblock would be copied.
> I think that it should be very unusual that installboot would be run
> interactively.  That is really no excuse for making it only slightly
Indeed: it should mostly be run behind the scenes e.g. by live upgrade, but
obviously there are scenarios where it is necessary (like this one).
> smarter than dd, but it might be hard to justify changes unless some
> kind person were to submit a bug with an improved implementation
> (would make a good short project for someone :-)
The problem here might be that an improved implementation would probably
mean an incompatible change (like doing away with the explicit bootblk
argument).

Unfortunately, I''ve too many other issues on my plate right now to
attack
this one.

	Rainer

-----------------------------------------------------------------------------
Rainer Orth, Faculty of Technology, Bielefeld University

Rainer,

Sorry for your trouble.

I''m updating the installboot example in the ZFS Admin Guide with the
-F zfs syntax now. We''ll fix the installboot man page as well.

Mark, I don''t have an x86 system to test right now, can you send
me the correct installgrub syntax for booting a ZFS file system?

Thanks,

Cindy

Rainer Orth wrote:
> Rainer Orth <ro at techfak.uni-bielefeld.de> writes:
> 
> 
>>>instlalboot on the new disk and see if that fixes it.
>>
>>Unfortunately, it didn''t.  Reconsidering now, I see that I ran
installboot
>>against slice 0 (reduced by 1 sector as required by CR 6680633) instead
of
>>slice 2 (whole disk).  Doing so doesn''t fix the problem either,
though.
> 
> 
> I''ve found out what the problem was: I didn''t specify the
-F zfs option to
> installboot, so only half of the ZFS bootblock was written.  This is a
> combination of two documentation bugs and a terrible interface:
> 
> * With the introduction of zfs boot, installboot got a new -F
<fstype>
>   option.  Unfortunately, this is documented neither on installboot(1M)
>   (which wasn''t update at all, it seems) nor in the ZFS Admin
Guide
>   (p.80, workaround for CR 6668666).
> 
> * Apart from that, I''ve never understood why it is necessary to
specify the
>   full path to the bootblock to installboot like this
> 
> installboot /usr/platform/`uname -i`/lib/fs/<fstype>/bootblk
/dev/rdsk/c0t0d0s0
> 
>   It would be far easier to just specify the fstype (or even let
>   installboot figure that out by itself using fstyp) than having to give
>   the full pathname.  In that case, installboot could just dd the whole
>   bootblk file instead of hardcoding the block counts for the different
>   filesystem types (probably to avoid corrupting the filesystem if the user
>   gives a file that is not a bootblock).
> 
> Overall, a terrible mess ;-(
> 
> Regards.
> 	Rainer
>

Cindy,
> Sorry for your trouble.
no problem.
> I''m updating the installboot example in the ZFS Admin Guide with
the
> -F zfs syntax now. We''ll fix the installboot man page as well.
Great, thanks.

	Rainer

-----------------------------------------------------------------------------
Rainer Orth, Faculty of Technology, Bielefeld University

On Wed, 23 Jul 2008, Cindy.Swearingen at Sun.COM wrote:
> Rainer,
>
> Sorry for your trouble.
>
> I''m updating the installboot example in the ZFS Admin Guide with
the
> -F zfs syntax now. We''ll fix the installboot man page as well.
>
> Mark, I don''t have an x86 system to test right now, can you send
me the
> correct installgrub syntax for booting a ZFS file system?
It''s just installboot that had to change.  The installgrub CLI remains
the
same.



Regards,
markm

Rainer Orth wrote:
> Richard Elling writes:
>
>   
>>> I''ve found out what the problem was: I didn''t
specify the -F zfs option to
>>> installboot, so only half of the ZFS bootblock was written.  This
is a
>>> combination of two documentation bugs and a terrible interface:
>>>   
>>>       
>> Mainly because there is no -F option?
>>     
>
> Huh?  From /usr/sbin/installboot:
>   
Which build you see this? It isn''t in the online source
browser or b93... there might be another issue lurking here...
> COUNT=15
>
> while getopts F: a; do
>         case $a in
>         F) case $OPTARG in
>            ufs) COUNT=15;;
>            hsfs) COUNT=15;;
>            zfs) COUNT=31;;
>            *) away 1 "$OPTARG: Unknown fstype";;
>            esac;;
>
> Without -F zfs, only part of the zfs bootblock would be copied.
>
>   
>> I think that it should be very unusual that installboot would be run
>> interactively.  That is really no excuse for making it only slightly
>>     
>
> Indeed: it should mostly be run behind the scenes e.g. by live upgrade, but
> obviously there are scenarios where it is necessary (like this one).
>
>   
>> smarter than dd, but it might be hard to justify changes unless some
>> kind person were to submit a bug with an improved implementation
>> (would make a good short project for someone :-)
>>     
>
> The problem here might be that an improved implementation would probably
> mean an incompatible change (like doing away with the explicit bootblk
> argument).
>
>   
Yes, though there are good reasons to use other bootblks.
 -- richard

Richard Elling wrote:
> Rainer Orth wrote:
>   
>> Richard Elling writes:
>>
>>   
>>     
>>>> I''ve found out what the problem was: I didn''t
specify the -F zfs option to
>>>> installboot, so only half of the ZFS bootblock was written. 
This is a
>>>> combination of two documentation bugs and a terrible interface:
>>>>   
>>>>       
>>>>         
>>> Mainly because there is no -F option?
>>>     
>>>       
>> Huh?  From /usr/sbin/installboot:
>>   
>>     
>
> Which build you see this? It isn''t in the online source
> browser or b93... there might be another issue lurking here...
>   
Never mind. I found it.  We''ll try to get this straightened out
in a way that will not confuse people who might not realize
that the x86 and sparc versions are so different.
 -- richard

On Wed, Jul 23, 2008 at 11:36 AM,  <Cindy.Swearingen at sun.com>
wrote:
> Rainer,
>
> Sorry for your trouble.
>
> I''m updating the installboot example in the ZFS Admin Guide with
the
> -F zfs syntax now. We''ll fix the installboot man page as well.
Perhaps it also deserves a mention in the FAQ somewhere near
http://opensolaris.org/os/community/zfs/boot/zfsbootFAQ/#mirrorboot.

5. How do I attach a mirror to an existing ZFS root pool"?

Attach the second disk to form a mirror.  In this example, c1t1d0s0 is attached.

# zpool attach rpool c1t0d0s0 c1t1d0s0

Prior to build <TBD>, bug 6668666 causes the following
platform-dependent steps to also be needed:

On sparc systems:
# installboot -F zfs /usr/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c1t1d0s0

On x86 systems:
# ...

-- 
Mike Gerdts
http://mgerdts.blogspot.com/

Mike Gerdts wrote:
> On Wed, Jul 23, 2008 at 11:36 AM,  <Cindy.Swearingen at sun.com>
wrote:
>> Rainer,
>>
>> Sorry for your trouble.
>>
>> I''m updating the installboot example in the ZFS Admin Guide
with the
>> -F zfs syntax now. We''ll fix the installboot man page as well.
> 
> Perhaps it also deserves a mention in the FAQ somewhere near
> http://opensolaris.org/os/community/zfs/boot/zfsbootFAQ/#mirrorboot.
> 
> 5. How do I attach a mirror to an existing ZFS root pool"?
> 
> Attach the second disk to form a mirror.  In this example, c1t1d0s0 is
attached.
> 
> # zpool attach rpool c1t0d0s0 c1t1d0s0
> 
> Prior to build <TBD>, bug 6668666 causes the following
> platform-dependent steps to also be needed:
> 
> On sparc systems:
> # installboot -F zfs /usr/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c1t1d0s0
should be uname -m above I think.
and path to be:
# installboot -F zfs /platform/`uname -m`/lib/fs/zfs/bootblk as path for sparc.

others might correct me though
> 
> On x86 systems:
> # ...
>

Enda O''Connor ( Sun Micro Systems Ireland)
wrote:
> Mike Gerdts wrote:
>> On Wed, Jul 23, 2008 at 11:36 AM,  <Cindy.Swearingen at sun.com>
wrote:
>>> Rainer,
>>>
>>> Sorry for your trouble.
>>>
>>> I''m updating the installboot example in the ZFS Admin
Guide with the
>>> -F zfs syntax now. We''ll fix the installboot man page as
well.
>>
>> Perhaps it also deserves a mention in the FAQ somewhere near
>> http://opensolaris.org/os/community/zfs/boot/zfsbootFAQ/#mirrorboot.
>>
>> 5. How do I attach a mirror to an existing ZFS root pool"?
>>
>> Attach the second disk to form a mirror.  In this example, c1t1d0s0 is 
>> attached.
>>
>> # zpool attach rpool c1t0d0s0 c1t1d0s0
>>
>> Prior to build <TBD>, bug 6668666 causes the following
>> platform-dependent steps to also be needed:
>>
>> On sparc systems:
>> # installboot -F zfs /usr/`uname -i`/lib/fs/zfs/bootblk 
>> /dev/rdsk/c1t1d0s0
> 
> should be uname -m above I think.
> and path to be:
> # installboot -F zfs /platform/`uname -m`/lib/fs/zfs/bootblk as path for 
> sparc.
> 
> others might correct me though
> 
>>
>> On x86 systems:
>> # ...
meant to add that on x86 the following should do the trick ( again I''m
open to correction )

installgrub /boot/grub/stage1 /zfsroot/boot/grub/stage2 /dev/rdsk/c1t0d0s0

haven''t tested the z86 one though.

Enda
>>
> 
>

Hoping this is not too off topic.    Can anyone confirm you can break a mirrored
zfs root pool once formed.   I basically want to clone a boot drive, take it to
another piece of identical hardware and have two machines ( or more ).   I am
running indiana b93 on x86 hardware.     I have read that there are various bugs
with mirrored zfs root that prevent what I want to do.
 
 
This message posted from opensolaris.org

"Enda O''Connor ( Sun Micro Systems Ireland)" <Enda.Oconnor
at Sun.COM>
writes:
[..]
> meant to add that on x86 the following should do the trick ( again
I''m open to correction )
>
> installgrub /boot/grub/stage1 /zfsroot/boot/grub/stage2 /dev/rdsk/c1t0d0s0
>
> haven''t tested the z86 one though.
I used

installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c1t0d0s0

(i.e., no "/zfsroot")

Boyd

Boyd Adamson wrote:
> "Enda O''Connor ( Sun Micro Systems Ireland)"
<Enda.Oconnor at Sun.COM>
> writes:
> [..]
>> meant to add that on x86 the following should do the trick ( again
I''m open to correction )
>>
>> installgrub /boot/grub/stage1 /zfsroot/boot/grub/stage2
/dev/rdsk/c1t0d0s0
>>
>> haven''t tested the z86 one though.
> 
> I used
> 
> installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c1t0d0s0
> 
> (i.e., no "/zfsroot")
ah yes, testing does help all right :-)

Enda
> 
> Boyd
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss