I''ve always done a disksuite mirror of the boot disk. It''s been easry to do after the install in Solaris. WIth Linux I had do do it during the install. OpenSolaris 2008.05 didn''t give me an option. How do I add my 2nd drive to the boot zpool to make it a mirror? This message posted from opensolaris.org
Hi Tom, You need to use the zpool attach command, like this: # zpool attach pool-name disk1 disk2 Cindy Tom Buskey wrote:> I''ve always done a disksuite mirror of the boot disk. It''s been easry to do after the install in Solaris. WIth Linux I had do do it during the install. > > OpenSolaris 2008.05 didn''t give me an option. > > How do I add my 2nd drive to the boot zpool to make it a mirror? > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
It is also necessary to use either installboot (sparc) or installgrub (x86) to install the boot loader on the attached disk. It is a bug that this is not done automatically (6668666 - zpool command should put a bootblock on a disk added as a mirror of a root pool vdev) Lori Cindy.Swearingen at Sun.COM wrote:> Hi Tom, > > You need to use the zpool attach command, like this: > > # zpool attach pool-name disk1 disk2 > > Cindy > > Tom Buskey wrote: > >> I''ve always done a disksuite mirror of the boot disk. It''s been easry to do after the install in Solaris. WIth Linux I had do do it during the install. >> >> OpenSolaris 2008.05 didn''t give me an option. >> >> How do I add my 2nd drive to the boot zpool to make it a mirror? >> >> >> This message posted from opensolaris.org >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
I just tried that, but the installgrub keeps failing:
malachi at serveris:~# zpool status
pool: rpool
state: ONLINE
scrub: resilver completed after 0h1m with 0 errors on Sat Aug 2 01:44:55
2008
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c5t0d0s0 ONLINE 0 0 0
c5t1d0s0 ONLINE 0 0 0
errors: No known data errors
malachi at serveris:~# installgrub /boot/grub/stage1 /boot/grub/stage2
/dev/dsk/c5t1d0s0
cannot open/stat device /dev/dsk/c5t1d0s2
On Wed, May 21, 2008 at 3:19 PM, Lori Alt <Lori.Alt at sun.com> wrote:
>
> It is also necessary to use either installboot (sparc) or installgrub (x86)
> to install the boot loader on the attached disk. It is a bug that this
> is not done automatically (6668666 - zpool command should put a
> bootblock on a disk added as a mirror of a root pool vdev)
>
> Lori
>
> Cindy.Swearingen at Sun.COM wrote:
> > Hi Tom,
> >
> > You need to use the zpool attach command, like this:
> >
> > # zpool attach pool-name disk1 disk2
> >
> > Cindy
> >
> > Tom Buskey wrote:
> >
> >> I''ve always done a disksuite mirror of the boot disk.
It''s been easry
> to do after the install in Solaris. WIth Linux I had do do it during the
> install.
> >>
> >> OpenSolaris 2008.05 didn''t give me an option.
> >>
> >> How do I add my 2nd drive to the boot zpool to make it a mirror?
> >>
> >>
> >> This message posted from opensolaris.org
> >> _______________________________________________
> >> zfs-discuss mailing list
> >> zfs-discuss at opensolaris.org
> >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> >>
> > _______________________________________________
> > zfs-discuss mailing list
> > zfs-discuss at opensolaris.org
> > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> >
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080802/66755bda/attachment.html>
Malachi de ?lfweald wrote:> I just tried that, but the installgrub keeps failing: > > malachi at serveris:~# zpool status > pool: rpool > state: ONLINE > scrub: resilver completed after 0h1m with 0 errors on Sat Aug 2 > 01:44:55 2008 > config: > > NAME STATE READ WRITE CKSUM > rpool ONLINE 0 0 0 > mirror ONLINE 0 0 0 > c5t0d0s0 ONLINE 0 0 0 > c5t1d0s0 ONLINE 0 0 0 > > errors: No known data errors > malachi at serveris:~# installgrub /boot/grub/stage1 /boot/grub/stage2 > /dev/dsk/c5t1d0s0 > cannot open/stat device /dev/dsk/c5t1d0s2 >that should be /dev/rdsk/c5t1d0s2 Enda> > On Wed, May 21, 2008 at 3:19 PM, Lori Alt <Lori.Alt at sun.com > <mailto:Lori.Alt at sun.com>> wrote: > > > It is also necessary to use either installboot (sparc) or > installgrub (x86) > to install the boot loader on the attached disk. It is a bug that this > is not done automatically (6668666 - zpool command should put a > bootblock on a disk added as a mirror of a root pool vdev) > > Lori > > Cindy.Swearingen at Sun.COM wrote: > > Hi Tom, > > > > You need to use the zpool attach command, like this: > > > > # zpool attach pool-name disk1 disk2 > > > > Cindy > > > > Tom Buskey wrote: > > > >> I''ve always done a disksuite mirror of the boot disk. It''s been > easry to do after the install in Solaris. WIth Linux I had do do it > during the install. > >> > >> OpenSolaris 2008.05 didn''t give me an option. > >> > >> How do I add my 2nd drive to the boot zpool to make it a mirror? > >> > >> > >> This message posted from opensolaris.org <http://opensolaris.org> > >> _______________________________________________ > >> zfs-discuss mailing list > >> zfs-discuss at opensolaris.org <mailto:zfs-discuss at opensolaris.org> > >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > >> > > _______________________________________________ > > zfs-discuss mailing list > > zfs-discuss at opensolaris.org <mailto:zfs-discuss at opensolaris.org> > > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org <mailto:zfs-discuss at opensolaris.org> > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > > > ------------------------------------------------------------------------ > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
malachi at serveris:~# installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c5t1d0s2 raw device must be a root slice (not s2) and trying rdsk with s0 gave same error as before On Sat, Aug 2, 2008 at 2:02 AM, Enda O''Connor <Enda.Oconnor at sun.com> wrote:> Malachi de ?lfweald wrote: > >> I just tried that, but the installgrub keeps failing: >> >> malachi at serveris:~# zpool status >> pool: rpool >> state: ONLINE >> scrub: resilver completed after 0h1m with 0 errors on Sat Aug 2 01:44:55 >> 2008 >> config: >> >> NAME STATE READ WRITE CKSUM >> rpool ONLINE 0 0 0 >> mirror ONLINE 0 0 0 >> c5t0d0s0 ONLINE 0 0 0 >> c5t1d0s0 ONLINE 0 0 0 >> >> errors: No known data errors >> malachi at serveris:~# installgrub /boot/grub/stage1 /boot/grub/stage2 >> /dev/dsk/c5t1d0s0 >> cannot open/stat device /dev/dsk/c5t1d0s2 >> >> > that should be /dev/rdsk/c5t1d0s2 > > Enda > >> >> On Wed, May 21, 2008 at 3:19 PM, Lori Alt <Lori.Alt at sun.com <mailto: >> Lori.Alt at sun.com>> wrote: >> >> >> It is also necessary to use either installboot (sparc) or >> installgrub (x86) >> to install the boot loader on the attached disk. It is a bug that this >> is not done automatically (6668666 - zpool command should put a >> bootblock on a disk added as a mirror of a root pool vdev) >> >> Lori >> >> Cindy.Swearingen at Sun.COM wrote: >> > Hi Tom, >> > >> > You need to use the zpool attach command, like this: >> > >> > # zpool attach pool-name disk1 disk2 >> > >> > Cindy >> > >> > Tom Buskey wrote: >> > >> >> I''ve always done a disksuite mirror of the boot disk. It''s been >> easry to do after the install in Solaris. WIth Linux I had do do it >> during the install. >> >> >> >> OpenSolaris 2008.05 didn''t give me an option. >> >> >> >> How do I add my 2nd drive to the boot zpool to make it a mirror? >> >> >> >> >> >> This message posted from opensolaris.org <http://opensolaris.org> >> >> _______________________________________________ >> >> zfs-discuss mailing list >> >> zfs-discuss at opensolaris.org <mailto:zfs-discuss at opensolaris.org> >> >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> >> >> > _______________________________________________ >> > zfs-discuss mailing list >> > zfs-discuss at opensolaris.org <mailto:zfs-discuss at opensolaris.org> >> > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org <mailto:zfs-discuss at opensolaris.org> >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080802/cef753fa/attachment.html>
I ran into this as well. For some reason installgrub needs slice 2 to be the special "backup" slice that covers the whole disk, as in Solaris. You actually specify s0 on the command line since this is the location of the ZFS root, but installgrub will go away and try to access the whole disk using slice 2 for some reason. What I did to solve it was to use format to select the disk, then the "partition" option to create a slice 2 that started on cylinder 0 and ended on the final cylinder of the disk. Once I did that installgrub worked OK. You might also need to issue the command "disks" to get Solaris to update the disk links under /dev before you use installgrub. Andrew. This message posted from opensolaris.org
Have you verified that it will auto failover correctly if one is s0 and one is s2? On Sat, Aug 2, 2008 at 3:53 PM, andrew <andrum04 at gmail.com> wrote:> I ran into this as well. For some reason installgrub needs slice 2 to be > the special "backup" slice that covers the whole disk, as in Solaris. You > actually specify s0 on the command line since this is the location of the > ZFS root, but installgrub will go away and try to access the whole disk > using slice 2 for some reason. What I did to solve it was to use format to > select the disk, then the "partition" option to create a slice 2 that > started on cylinder 0 and ended on the final cylinder of the disk. Once I > did that installgrub worked OK. You might also need to issue the command > "disks" to get Solaris to update the disk links under /dev before you use > installgrub. > > Andrew. > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080802/0238392d/attachment.html>
The second disk doesn''t have the root pool on slice 2 - it is on slice 0 as with the first disk. All I did differently was to create a slice 2 covering the whole Solaris FDISK primary partition. If you then issue this command as before: installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/dsk/c5t1d0s0 (Note: slice ZERO) Then it will install grub onto that disk. You would need to ask someone else why it needs a slice 2 - I suspect that stage1 actually gets written to the first sector of the Solaris primary FDISK partition, hence it needs access to the "special" slice 2 to do that. Cheers Andrew. This message posted from opensolaris.org
OK, I''ve put up some screenshots and a copy of my menu.lst to clarify my setup: http://sites.google.com/site/solarium/zfs-screenshots Cheers Andrew. This message posted from opensolaris.org
I have to say, looking at that confuses me a little. How can the two disks be mirrored when the partition tables don''t match? On Sun, Aug 3, 2008 at 6:00 AM, andrew <andrum04 at gmail.com> wrote:> OK, I''ve put up some screenshots and a copy of my menu.lst to clarify my > setup: > > http://sites.google.com/site/solarium/zfs-screenshots > > Cheers > > Andrew. > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080803/f8919025/attachment.html>
Malachi de ?lfweald wrote:> I have to say, looking at that confuses me a little. How can the two > disks be mirrored when the partition tables don''t match?Welcome to ZFS! In traditional disk mirrors, disk A block 0 == disk B block 0 disk A block 1 == disk B block 1 ... disk A block N == disk B block N In a ZFS world, block 1 might be defective. So ZFS will reallocate the block somewhere else. This is great for reliable storage on unreliable devices (disks). It also relieves you from the expectation that the partition tables must match. And it also means that I can grow the pool size by replacing the mirror sides with larger devices. Life is good! Enjoy! -- richard
I took the brute force approach, but it was simple and passed the "boot from either" test: install on both, then mirror s0, and I''m reasonably confident identical disks will look the same ;-) This message posted from opensolaris.org
Richard Elling wrote:> Malachi de ?lfweald wrote: > >> I have to say, looking at that confuses me a little. How can the two >> disks be mirrored when the partition tables don''t match? >> > > Welcome to ZFS! In traditional disk mirrors, > disk A block 0 == disk B block 0 > disk A block 1 == disk B block 1 > ... > disk A block N == disk B block N > > In a ZFS world, block 1 might be defective. So ZFS will reallocate > the block somewhere else. This is great for reliable storage on > unreliable devices (disks). It also relieves you from the expectation > that the partition tables must match. And it also means that I can > grow the pool size by replacing the mirror sides with larger devices. > Life is good! Enjoy! >This functionality is not specific to ZFS. SVM mirroring also does not require partition tables to match and as in ZFS, one can grow the size of mirror by replacing larger size disks. -Sanjay> -- richard > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
So I spent some time trying to get the 2nd slice up on the 2nd disk... I did manage to finally get it on there by saving the partition table to format.dat and reformatting the 2nd disk using it, but as soon as I did the zpool attach, it wiped out the slice 2 again. I also tried the prtvtoc and fmthard after attaching, but that didn''t work either. Is there some specific steps I can follow to get the 2nd slice to stay on post-attach? Thanks, Malachi On Sun, Aug 3, 2008 at 6:00 AM, andrew <andrum04 at gmail.com> wrote:> OK, I''ve put up some screenshots and a copy of my menu.lst to clarify my > setup: > > http://sites.google.com/site/solarium/zfs-screenshots > > Cheers > > Andrew. > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080805/bda2db97/attachment.html>
Ok, here''s the end results... zfs attach rpool c5t0d0s0 c5t1d0s0: removes c5t1d0s2 zfs attach rpool c5t0d0s2 c5t1d0s2: says c5t0d0s2 is not in the pool zfs attach rpool c5t0d0s0 c5t1d0s2: says I have to force it because s0 and s2 overlap zfs attach -f rpool c5t0d0s0 c5t1d0s2: partition table now matches - onto next step installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c5t1d0s2 raw device must be a root slice (not s2) installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c5t1d0s0 stage1 written to partition 0 sector 0 (abs 16065) stage2 written to partition 0, 260 sectors starting at 50 (abs 16115) I *think* this means it is good to go? What is the easiest way to test it? Malachi On Tue, Aug 5, 2008 at 9:12 PM, Ellis, Mike <Mike.Ellis at fmr.com> wrote:> yep... > > when you''re "source pool" is a slice, you better add a slice as a target > pool if you want to have that slice on the target side remain. > > should work fine that way, > > -- MikeE > > ------------------------------ > *From:* Malachi de ?lfweald [mailto:malachid at gmail.com] > *Sent:* Wednesday, August 06, 2008 12:11 AM > *To:* Ellis, Mike > *Subject:* Re: [zfs-discuss] ZFS boot mirror > > Hmmm. I tried c5t1d0 which gave an error and c5t1d0s0 which is what > overwrote it. Maybe I should try mirroring c5t0d0s0 to c5t1d0s2? > > On Tue, Aug 5, 2008 at 8:53 PM, Ellis, Mike <Mike.Ellis at fmr.com> wrote: > >> did you zpool attach "the whole disk" or the specific slice you >> prepared? >> >> -- MikeE >> >> ------------------------------ >> *From:* zfs-discuss-bounces at opensolaris.org [mailto: >> zfs-discuss-bounces at opensolaris.org] *On Behalf Of *Malachi de ?lfweald >> *Sent:* Tuesday, August 05, 2008 11:42 PM >> *To:* andrew >> *Cc:* zfs-discuss at opensolaris.org >> *Subject:* Re: [zfs-discuss] ZFS boot mirror >> >> So I spent some time trying to get the 2nd slice up on the 2nd disk... >> I did manage to finally get it on there by saving the partition table to >> format.dat and reformatting the 2nd disk using it, but as soon as I did the >> zpool attach, it wiped out the slice 2 again. I also tried the prtvtoc and >> fmthard after attaching, but that didn''t work either. >> >> Is there some specific steps I can follow to get the 2nd slice to stay on >> post-attach? >> >> Thanks, >> Malachi >> >> On Sun, Aug 3, 2008 at 6:00 AM, andrew <andrum04 at gmail.com> wrote: >> >>> OK, I''ve put up some screenshots and a copy of my menu.lst to clarify my >>> setup: >>> >>> http://sites.google.com/site/solarium/zfs-screenshots >>> >>> Cheers >>> >>> Andrew. >>> >>> >>> This message posted from opensolaris.org >>> _______________________________________________ >>> zfs-discuss mailing list >>> zfs-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>> >> >> >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080805/55bb7e87/attachment.html>
Sounds like you''ve got an EFI label on the second disk. Can you run "format", select the second disk, then enter "fdisk" then "print" and post the output here? Thanks Andrew. This message posted from opensolaris.org
It looks like we finally got it working. The log of what Mike had me do to fix it is here<http://malsserver.blogspot.com/2008/08/mirroring-resolved-correct-way.html>in case anyone else runs into this. Thanks to everyone who helped with this. Thanks again! Mal On Tue, Aug 5, 2008 at 9:40 PM, andrew <andrum04 at gmail.com> wrote:> Sounds like you''ve got an EFI label on the second disk. Can you run > "format", select the second disk, then enter "fdisk" then "print" and post > the output here? > > Thanks > > Andrew. > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080805/4f5e4aa0/attachment.html>