I''m setting up a new X4500 Thumper, and noticed suggestions/blogs for setting up two boot disks as a zfs rpool mirror during installation. But I can''t seem to find instructions/examples for how to do this using google, the blogs, or the Sun docs for X4500. Can anyone share some instructions for setting up the rpool mirror of the boot disks during the Solaris Nevada (SXCE) install? Thanks, Neal
On Tue, 17 Mar 2009, Neal Pollack wrote:> Can anyone share some instructions for setting up the rpool mirror of > the boot disks during the Solaris Nevada (SXCE) install?You''ll need to use the text-based installer, and in there you choose two the two bootable disks instead of just one. They''re automatically mirrored. Regards, markm
Cindy.Swearingen at Sun.COM
2009-Mar-17 19:32 UTC
[zfs-discuss] How do I "mirror" zfs rpool, x4500?
Neal, You''ll need to use the text-based initial install option. The steps for configuring a ZFS root pool during an initial install are covered here: http://opensolaris.org/os/community/zfs/docs/ Page 114: Example 4?1 Initial Installation of a Bootable ZFS Root File System Step 3, you''ll be presented with the disks to be selected as in previous releases. So, for example, to select the boot disks on the Thumper, select both of them: [x] c5t0d0 [x] c4t0d0 . . . On our lab Thumper, they are c5t0 and c4t0. Cindy Neal Pollack wrote:> I''m setting up a new X4500 Thumper, and noticed suggestions/blogs > for setting up two boot disks as a zfs rpool mirror during installation. > But I can''t seem to find instructions/examples for how to do this using > google, the blogs, or the Sun docs for X4500. > > Can anyone share some instructions for setting up the rpool mirror > of the boot disks during the Solaris Nevada (SXCE) install? > > Thanks, > > Neal > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On 17-Mar-09, at 3:32 PM, Cindy.Swearingen at Sun.COM wrote:> Neal, > > You''ll need to use the text-based initial install option. > The steps for configuring a ZFS root pool during an initial > install are covered here: > > http://opensolaris.org/os/community/zfs/docs/ > > Page 114: > > Example 4?1 Initial Installation of a Bootable ZFS Root File System > > Step 3, you''ll be presented with the disks to be selected as in > previous releases. So, for example, to select the boot disks on the > Thumper, > select both of them:Right, but what if you didn''t realise on that screen that you needed to select both to make a mirror? The wording isn''t very explicit, in my opinion. Yesterday I did my first Solaris 10 ZFS root install and didn''t interpret this screen correctly. I chose one disk, so I''m the OP''s situation and want to set up the mirror retrospectively. I''m using an X2100. Unfortunately when I try to zpool attach, I get a Device busy error on the 2nd drive. But probably I''m making a n00b error. --Toby> > [x] c5t0d0 > [x] c4t0d0 > . > . > . > > > On our lab Thumper, they are c5t0 and c4t0. > > Cindy > > Neal Pollack wrote: >> I''m setting up a new X4500 Thumper, and noticed suggestions/blogs >> for setting up two boot disks as a zfs rpool mirror during >> installation. >> But I can''t seem to find instructions/examples for how to do this >> using >> google, the blogs, or the Sun docs for X4500. >> Can anyone share some instructions for setting up the rpool mirror >> of the boot disks during the Solaris Nevada (SXCE) install? >> Thanks, >> Neal >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
+------------------------------------------------------------------------------ | On 2009-03-17 16:13:27, Toby Thain wrote: | | Right, but what if you didn''t realise on that screen that you needed | to select both to make a mirror? The wording isn''t very explicit, in | my opinion. Yesterday I did my first Solaris 10 ZFS root install and | didn''t interpret this screen correctly. I chose one disk, so I''m the | OP''s situation and want to set up the mirror retrospectively. | | I''m using an X2100. Unfortunately when I try to zpool attach, I get a | Device busy error on the 2nd drive. But probably I''m making a n00b | error. Use format(1M) to ensure the second disk (c1t1d0) is formatted as 100% Solaris. Then mirror the VTOC from the first (zfsroot) disk to the second: # prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2 # zpool attach -f rpool c1t0d0s0 c1t1d0s0 # zpool status -v -- bda Cyberpunk is dead. Long live cyberpunk. http://mirrorshades.org
On 17 Mar, 2009, at 16.21, Bryan Allen wrote:> Then mirror the VTOC from the first (zfsroot) disk to the second: > > # prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2 > # zpool attach -f rpool c1t0d0s0 c1t1d0s0 > # zpool status -vAnd then you''ll still need to run installgrub to put grub on the mirror. That''s not yet automatically done. Regards, markm -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090317/0d87ba14/attachment.html>
+------------------------------------------------------------------------------ | On 2009-03-17 16:37:25, Mark J Musante wrote: | | >Then mirror the VTOC from the first (zfsroot) disk to the second: | > | ># prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2 | ># zpool attach -f rpool c1t0d0s0 c1t1d0s0 | ># zpool status -v | | And then you''ll still need to run installgrub to put grub on the | mirror. That''s not yet automatically done. Ah, yes, thanks. installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c1t1d0s0 Knew I forgot something. Got distracted by local boom. -- bda Cyberpunk is dead. Long live cyberpunk. http://mirrorshades.org
On 03/17/09 12:32 PM, Cindy.Swearingen at Sun.COM wrote:> Neal, > > You''ll need to use the text-based initial install option. > The steps for configuring a ZFS root pool during an initial > install are covered here: > > http://opensolaris.org/os/community/zfs/docs/ > > Page 114: > > Example 4?1 Initial Installation of a Bootable ZFS Root File System > > Step 3, you''ll be presented with the disks to be selected as in > previous releases. So, for example, to select the boot disks on the > Thumper, > select both of them: > > [x] c5t0d0 > [x] c4t0d0Why have the controller numbers/mappings changed between Solaris 10 and Solaris Nevada? I just installed Solaris Nevada 110 to see what it would do. Thank you, and I now understand that to find the disk name, like above c5t0d0 for physical slot 0 on X4500, I can use "cfgadm | grep sata3/0" I also now understand that in the installer screens, I can select 2 disks and they will become a mirrored root zpool. What I do not understand, is that on Solaris Nevada 110, the x4500 Thumper physical disk slots 0 and 1 are labeled as controller 3 and not controller 5. For example; # cfgadm | grep sata3/0 sata3/0::dsk/c3t0d0 disk connected configured ok # cfgadm | grep sata3/4 sata3/4::dsk/c3t4d0 disk connected configured ok # uname -a SunOS zcube-1 5.11 snv_110 i86pc i386 i86pc # Of course, that means I shold stay away from all the X4500 and ZFS docs if I run Solaris Nevada on an X4500? Any ideas why the mapping is not matching s10 or the docs? Cheers, Neal> . > . > . > > > On our lab Thumper, they are c5t0 and c4t0. > > Cindy > > Neal Pollack wrote: >> I''m setting up a new X4500 Thumper, and noticed suggestions/blogs >> for setting up two boot disks as a zfs rpool mirror during installation. >> But I can''t seem to find instructions/examples for how to do this using >> google, the blogs, or the Sun docs for X4500. >> >> Can anyone share some instructions for setting up the rpool mirror >> of the boot disks during the Solaris Nevada (SXCE) install? >> >> Thanks, >> >> Neal >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Tue, Mar 17, 2009 at 03:51:25PM -0700, Neal Pollack wrote:> >Step 3, you''ll be presented with the disks to be selected as in > >previous releases. So, for example, to select the boot disks on the > >Thumper, > >select both of them: > > > >[x] c5t0d0 > >[x] c4t0d0 > > Why have the controller numbers/mappings changed between Solaris 10 > and Solaris Nevada? I just installed Solaris Nevada 110 to see what > it would do. Thank you, and I now understand that to find the disk > name, like above c5t0d0 for physical slot 0 on X4500, I can use > "cfgadm | grep sata3/0"> I also now understand that in the installer screens, I can select 2 > disks and they > will become a mirrored root zpool. >> What I do not understand, is that on Solaris Nevada 110, the x4500 > Thumper physical > disk slots 0 and 1 are labeled as controller 3 and not controller 5. > For example; > > # cfgadm | grep sata3/0 > sata3/0::dsk/c3t0d0 disk connected configured ok > # cfgadm | grep sata3/4 > sata3/4::dsk/c3t4d0 disk connected configured ok > # uname -a > SunOS zcube-1 5.11 snv_110 i86pc i386 i86pcThe numberings are not pre-set, and probably have nothing to do with Solaris 10 vs Nevada (or ZFS). Controller numberings are sequential as they are discovered by the OS. So different probe order, post-boot hardware installations, or when drivers get installed can case the number assigned to be different on different machines.> Of course, that means I shold stay away from all the X4500 and ZFS docs if > I run Solaris Nevada on an X4500?Why would that be? It doesn''t claim that there will be a particular mapping for a particular X4500.> Any ideas why the mapping is not matching s10 or the docs?As far as I read, the docs are giving you an example. They''re not declaring that yours will be the same. -- Darren
On Wed, Mar 18, 2009 at 10:59 AM, A Darren Dunham <ddunham at taos.com> wrote:> On Tue, Mar 17, 2009 at 03:51:25PM -0700, Neal Pollack wrote: > > >Step 3, you''ll be presented with the disks to be selected as in > > >previous releases. So, for example, to select the boot disks on the > > >Thumper, > > >select both of them: > > > > > >[x] c5t0d0 > > >[x] c4t0d0 > > > > Why have the controller numbers/mappings changed between Solaris 10 > > and Solaris Nevada? I just installed Solaris Nevada 110 to see what > > it would do. Thank you, and I now understand that to find the disk > > name, like above c5t0d0 for physical slot 0 on X4500, I can use > > "cfgadm | grep sata3/0" > > > I also now understand that in the installer screens, I can select 2 > > disks and they > > will become a mirrored root zpool. > > > > > What I do not understand, is that on Solaris Nevada 110, the x4500 > > Thumper physical > > disk slots 0 and 1 are labeled as controller 3 and not controller 5. > > For example; > > > > # cfgadm | grep sata3/0 > > sata3/0::dsk/c3t0d0 disk connected configured ok > > # cfgadm | grep sata3/4 > > sata3/4::dsk/c3t4d0 disk connected configured ok > > # uname -a > > SunOS zcube-1 5.11 snv_110 i86pc i386 i86pc > > The numberings are not pre-set, and probably have nothing to do with > Solaris 10 vs Nevada (or ZFS). > > Controller numberings are sequential as they are discovered by the OS. > So different probe order, post-boot hardware installations, or when > drivers get installed can case the number assigned to be different on > different machines. > > > Of course, that means I shold stay away from all the X4500 and ZFS docs > if > > I run Solaris Nevada on an X4500? > > Why would that be? It doesn''t claim that there will be a particular > mapping for a particular X4500. > > > Any ideas why the mapping is not matching s10 or the docs? > > As far as I read, the docs are giving you an example. They''re not > declaring that yours will be the same. >Just an observation, but it sort of defeats the purpose of buying sun hardware with sun software if you can''t even get a "this is how your drives will map" out of the deal... --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090318/b38f1abd/attachment.html>
Tim wrote:> > Just an observation, but it sort of defeats the purpose of buying sun > hardware with sun software if you can''t even get a "this is how your > drives will map" out of the deal...Sun could fix that, but would you really want a replacement for BIOS? -- richard
+------------------------------------------------------------------------------ | On 2009-03-18 10:14:26, Richard Elling wrote: | | >Just an observation, but it sort of defeats the purpose of buying sun | >hardware with sun software if you can''t even get a "this is how your | >drives will map" out of the deal... | | Sun could fix that, but would you really want a replacement for BIOS? Well, actually... :) -- bda Cyberpunk is dead. Long live cyberpunk. http://mirrorshades.org
On Wed, Mar 18, 2009 at 12:14 PM, Richard Elling <richard.elling at gmail.com>wrote:> Tim wrote: > >> >> Just an observation, but it sort of defeats the purpose of buying sun >> hardware with sun software if you can''t even get a "this is how your drives >> will map" out of the deal... >> > > Sun could fix that, but would you really want a replacement for BIOS? > -- richard > >Yes, I really would. I also have a hard time believing BIOS is the issue. I have a 7110 sitting directly below an x4240 in one of my racks... the 7110 has no issues reporting disks properly. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090318/63aacc78/attachment.html>
On 03/18/09 10:43 AM, Tim wrote:> On Wed, Mar 18, 2009 at 12:14 PM, Richard Elling > <richard.elling at gmail.com <mailto:richard.elling at gmail.com>> wrote: > > Tim wrote: > > > Just an observation, but it sort of defeats the purpose of > buying sun hardware with sun software if you can''t even get a > "this is how your drives will map" out of the deal... > > > Sun could fix that, but would you really want a replacement for BIOS? > -- richard > > > Yes, I really would. I also have a hard time believing BIOS is the > issue. I have a 7110 sitting directly below an x4240 in one of my > racks... the 7110 has no issues reporting disks properly.BIOS is indeed an issue. In many x86/x64 PC architecture designs, and the current enumeration design of Solaris, if you add controller cards, or move a controller card, after a previous OS installation, then the controller numbers and ordering changes on all the devices. ZFS apparently does not care, but UFS would, since bios designates a specific disk to boot from, and the OS would have a specific boot path including a controller number such as /dev/dsk/c3t4d0s0 that could change, hence no longer boot. Getting to EFI firmware, dumping BIOS, and redesigning the Solaris device enumeration framework would make things a little more flexible in that type of scenario.> > --Tim > ------------------------------------------------------------------------ > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090318/cd445a72/attachment.html>
On Wed, Mar 18, 2009 at 12:49 PM, Neal Pollack <Neal.Pollack at sun.com> wrote:> On 03/18/09 10:43 AM, Tim wrote: > > On Wed, Mar 18, 2009 at 12:14 PM, Richard Elling <richard.elling at gmail.com > > wrote: > >> Tim wrote: >> >>> >>> Just an observation, but it sort of defeats the purpose of buying sun >>> hardware with sun software if you can''t even get a "this is how your drives >>> will map" out of the deal... >>> >> >> Sun could fix that, but would you really want a replacement for BIOS? >> -- richard >> >> > Yes, I really would. I also have a hard time believing BIOS is the issue. > I have a 7110 sitting directly below an x4240 in one of my racks... the 7110 > has no issues reporting disks properly. > > > BIOS is indeed an issue. In many x86/x64 PC architecture designs, and the > current enumeration design of Solaris, > if you add controller cards, or move a controller card, after a previous OS > installation, then the controller numbers > and ordering changes on all the devices. ZFS apparently does not care, but > UFS would, since bios designates a specific > disk to boot from, and the OS would have a specific boot path including a > controller number such as > /dev/dsk/c3t4d0s0 that could change, hence no longer boot. > > Getting to EFI firmware, dumping BIOS, and redesigning the Solaris device > enumeration framework would > make things a little more flexible in that type of scenario. > >How does any of that affect an x4500 with onboard controllers that can''t ever be moved? --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090318/b5238a8b/attachment.html>
Hi Tim, Tim wrote:> > How does any of that affect an x4500 with onboard controllers that can''t > ever be moved?Well, consider one box being installed from CD (external USB-CD) and another one which is jumpstarted via the network. The results usually are two different boot device names :( Q: Is there an easy way to reset this without breaking everything? Cheers Carsten
On Wed, Mar 18, 2009 at 07:13:41PM +0100, Carsten Aulbert wrote:> Well, consider one box being installed from CD (external USB-CD) and > another one which is jumpstarted via the network. The results usually > are two different boot device names :( > > Q: Is there an easy way to reset this without breaking everything?The mapping should be in /dev/cfg (and possibly portions of /etc/path_to_inst). In the old days I''d say that changing a non-boot controller there should be enough. I''m not sure if anything in the boot archive needs to be changed as well. -- Darren
On 03/18/09 11:09 AM, Tim wrote:> > > On Wed, Mar 18, 2009 at 12:49 PM, Neal Pollack <Neal.Pollack at sun.com > <mailto:Neal.Pollack at sun.com>> wrote: > > On 03/18/09 10:43 AM, Tim wrote: >> On Wed, Mar 18, 2009 at 12:14 PM, Richard Elling >> <richard.elling at gmail.com <mailto:richard.elling at gmail.com>> wrote: >> >> Tim wrote: >> >> >> Just an observation, but it sort of defeats the purpose >> of buying sun hardware with sun software if you can''t >> even get a "this is how your drives will map" out of the >> deal... >> >> >> Sun could fix that, but would you really want a replacement >> for BIOS? >> -- richard >> >> >> Yes, I really would. I also have a hard time believing BIOS is >> the issue. I have a 7110 sitting directly below an x4240 in one >> of my racks... the 7110 has no issues reporting disks properly. > > BIOS is indeed an issue. In many x86/x64 PC architecture designs, > and the current enumeration design of Solaris, > if you add controller cards, or move a controller card, after a > previous OS installation, then the controller numbers > and ordering changes on all the devices. ZFS apparently does not > care, but UFS would, since bios designates a specific > disk to boot from, and the OS would have a specific boot path > including a controller number such as > /dev/dsk/c3t4d0s0 that could change, hence no longer boot. > > Getting to EFI firmware, dumping BIOS, and redesigning the Solaris > device enumeration framework would > make things a little more flexible in that type of scenario. > > > > How does any of that affect an x4500 with onboard controllers that > can''t ever be moved?Stick a fiber channel controller card into your x4500 PCI slot, then go back and look at your controller numbering, even for the built-in disk controller chips. Here is the cfgadm output for an X4500 that I set up yesterday. Notice that the first two controller numbers are for the fibre channel devices, and then notice that the disk controller numbers no longer match your documentation, or your blogs about suggested configuration; $ cat zcube1.txt Ap_Id Type Receptacle Occupant Condition c6 fc connected unconfigured unknown c7 fc connected unconfigured unknown sata0/0::dsk/c0t0d0 disk connected configured ok sata0/1::dsk/c0t1d0 disk connected configured ok sata0/2::dsk/c0t2d0 disk connected configured ok sata0/3::dsk/c0t3d0 disk connected configured ok sata0/4::dsk/c0t4d0 disk connected configured ok sata0/5::dsk/c0t5d0 disk connected configured ok sata0/6::dsk/c0t6d0 disk connected configured ok sata0/7::dsk/c0t7d0 disk connected configured ok sata1/0::dsk/c1t0d0 disk connected configured ok sata1/1::dsk/c1t1d0 disk connected configured ok sata1/2::dsk/c1t2d0 disk connected configured ok sata1/3::dsk/c1t3d0 disk connected configured ok sata1/4::dsk/c1t4d0 disk connected configured ok sata1/5::dsk/c1t5d0 disk connected configured ok sata1/6::dsk/c1t6d0 disk connected configured ok sata1/7::dsk/c1t7d0 disk connected configured ok sata2/0::dsk/c2t0d0 disk connected configured ok sata2/1::dsk/c2t1d0 disk connected configured ok sata2/2::dsk/c2t2d0 disk connected configured ok sata2/3::dsk/c2t3d0 disk connected configured ok sata2/4::dsk/c2t4d0 disk connected configured ok sata2/5::dsk/c2t5d0 disk connected configured ok sata2/6::dsk/c2t6d0 disk connected configured ok sata2/7::dsk/c2t7d0 disk connected configured ok sata3/0::dsk/c3t0d0 disk connected configured ok <<-- Boot disk, slot 0 sata3/1::dsk/c3t1d0 disk connected configured ok sata3/2::dsk/c3t2d0 disk connected configured ok sata3/3::dsk/c3t3d0 disk connected configured ok sata3/4::dsk/c3t4d0 disk connected configured ok <<-- Boot disk, slot 1 sata3/5::dsk/c3t5d0 disk connected configured ok sata3/6::dsk/c3t6d0 disk connected configured ok sata3/7::dsk/c3t7d0 disk connected configured ok sata4/0::dsk/c4t0d0 disk connected configured ok sata4/1::dsk/c4t1d0 disk connected configured ok sata4/2::dsk/c4t2d0 disk connected configured ok sata4/3::dsk/c4t3d0 disk connected configured ok sata4/4::dsk/c4t4d0 disk connected configured ok sata4/5::dsk/c4t5d0 disk connected configured ok sata4/6::dsk/c4t6d0 disk connected configured ok sata4/7::dsk/c4t7d0 disk connected configured ok sata5/0::dsk/c5t0d0 disk connected configured ok sata5/1::dsk/c5t1d0 disk connected configured ok sata5/2::dsk/c5t2d0 disk connected configured ok sata5/3::dsk/c5t3d0 disk connected configured ok sata5/4::dsk/c5t4d0 disk connected configured ok sata5/5::dsk/c5t5d0 disk connected configured ok sata5/6::dsk/c5t6d0 disk connected configured ok sata5/7::dsk/c5t7d0 disk connected configured ok usb0/1 usb-device connected configured ok usb0/2 unknown empty unconfigured ok usb0/3 unknown empty unconfigured ok usb1/1 unknown empty unconfigured ok usb1/2 unknown empty unconfigured ok> > --Tim-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090318/76ef53ff/attachment.html>