I''m replacing the disk with my rpool with a mirrored pool, and wondering how best to do that. The disk I''m replacing is partitioned with root on s0, swap on s1 and boot on s8, which is what the original 2008.05 installer created for me. I''ve partitioned the new disk in the same way and am now running zpool attach -f rpool c5d0s0 c9t1d0s0 to mirror the root. The -f is to stop zpool whining about s0 overlapping s2. But what do I do with that swap slice? Should I ditch it and create an rpool/swap area? Do I still need a boot slice? What''s the recommended practice here? Cheers, Chris
On Wed, 26 Nov 2008 12:51:04 +0000 Chris Ridd <chrisridd at mac.com> wrote:> I''m replacing the disk with my rpool with a mirrored pool, and > wondering how best to do that. > > The disk I''m replacing is partitioned with root on s0, swap on s1 > and boot on s8, which is what the original 2008.05 installer created > for me.Are you sure about this? OS2008-05 uses the whole disk as a ZFS and within that rpool creates the seperate filesystems (swap/dump,..) I''ve never seen a ZFS system on seperate slices. Slices are things from the past ;-)> to mirror the root. The -f is to stop zpool whining about s0 > overlapping s2.If I use a disk for a root pool I create just one slice on it (s0). Nothing else. This is needed because booting of EFI labeled disks is not spuurted (yet).> But what do I do with that swap slice? Should I ditch it and create > an rpool/swap area? Do I still need a boot slice?ALL parts are created within the one rpool. -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv101 ++ + All that''s really worth doing is what we do for others (Lewis Carrol)
On 26 Nov 2008, at 13:12, dick hoogendijk wrote:> On Wed, 26 Nov 2008 12:51:04 +0000 > Chris Ridd <chrisridd at mac.com> wrote: > >> I''m replacing the disk with my rpool with a mirrored pool, and >> wondering how best to do that. >> >> The disk I''m replacing is partitioned with root on s0, swap on s1 >> and boot on s8, which is what the original 2008.05 installer created >> for me. > > Are you sure about this? OS2008-05 uses the whole disk as a ZFS and > within that rpool creates the seperate filesystems (swap/dump,..)Yep.> I''ve never seen a ZFS system on seperate slices. Slices are things > from > the past ;-)Maybe this is just a hangover from my original 2008.05 install?>> to mirror the root. The -f is to stop zpool whining about s0 >> overlapping s2. > > If I use a disk for a root pool I create just one slice on it (s0). > Nothing else. This is needed because booting of EFI labeled disks is > not spuurted (yet).Nod, I had to use format -e to force an SMI label.>> But what do I do with that swap slice? Should I ditch it and create >> an rpool/swap area? Do I still need a boot slice? > > ALL parts are created within the one rpool.Hm, so it might be better to do a new install onto the new disk with whatever slices the installer wants to set up, and then migrate the filesystems across from the old rpool. So where does installgrub put the boot bits? Cheers, Chris
Chris Ridd wrote:> On 26 Nov 2008, at 13:12, dick hoogendijk wrote: > > >> On Wed, 26 Nov 2008 12:51:04 +0000 >> Chris Ridd <chrisridd at mac.com> wrote: >> >> >>> I''m replacing the disk with my rpool with a mirrored pool, and >>> wondering how best to do that. >>> >>> The disk I''m replacing is partitioned with root on s0, swap on s1 >>> and boot on s8, which is what the original 2008.05 installer created >>> for me. >>> >> Are you sure about this? OS2008-05 uses the whole disk as a ZFS and >> within that rpool creates the seperate filesystems (swap/dump,..) >> > > Yep. > > >> I''ve never seen a ZFS system on seperate slices. Slices are things >> from >> the past ;-) >> > > Maybe this is just a hangover from my original 2008.05 install? > > >>> to mirror the root. The -f is to stop zpool whining about s0 >>> overlapping s2. >>> >> If I use a disk for a root pool I create just one slice on it (s0). >> Nothing else. This is needed because booting of EFI labeled disks is >> not spuurted (yet). >> > > Nod, I had to use format -e to force an SMI label. > > >>> But what do I do with that swap slice? Should I ditch it and create >>> an rpool/swap area? Do I still need a boot slice? >>> >> ALL parts are created within the one rpool. >> > > Hm, so it might be better to do a new install onto the new disk with > whatever slices the installer wants to set up, and then migrate the > filesystems across from the old rpool. > > So where does installgrub put the boot bits? >To clear up some confusion.. This is from a default indiana install format -e verify.. Part Tag Flag Cylinders Size Blocks 0 root wm 262 - 19453 147.02GB (19192/0/0) 308319480 1 swap wu 1 - 261 2.00GB (261/0/0) 4192965 So clearly root is on 0 and swap is on 1.. You *can not* install zfs to a whole disk and expect it to boot.. As for installing grub so it''s correct.. Choice 1) Best choice imho /zfsroot/boot/solaris/bin/update_grub -R /zfsroot Choice 2) (risky) installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c4t0d0s0 If you''re doing any sort of install by hand I have extensive notes on this I''m willing to share if you pm me. There''s a number of bugs you can easily hit/avoid as well. So be careful with have the root slice start at 1 for example. Good luck ./C
dick hoogendijk wrote:> On Wed, 26 Nov 2008 12:51:04 +0000 > Chris Ridd <chrisridd at mac.com> wrote: > >> I''m replacing the disk with my rpool with a mirrored pool, and >> wondering how best to do that. >> >> The disk I''m replacing is partitioned with root on s0, swap on s1 >> and boot on s8, which is what the original 2008.05 installer created >> for me. > > Are you sure about this? OS2008-05 uses the whole disk as a ZFS and > within that rpool creates the seperate filesystems (swap/dump,..) >The original builds of OpenSolaris 2008.05 did NOT use the whole disk. They created a separate swap slice in the Solaris VTOC in side the SOLARIS2 fdisk partition and put the pool on the rest. That swap slice was configured as swap and dump.> I''ve never seen a ZFS system on seperate slices. Slices are things from > the past ;-)Unfortunately not the case for ZFS pools that are to be booted from. This is because we can''t boot from an EFI labelled disk. Well maybe MacOS X hardware could but general x86 systems are still legacy BIOS and not EFI loaders. Even the current 2008.11 development builds still use a Solaris VTOC it is just that they make the slice the full size of the SOLARIS2 fdisk partition so it looks like it is using the whole disk.>> to mirror the root. The -f is to stop zpool whining about s0 >> overlapping s2. > > If I use a disk for a root pool I create just one slice on it (s0). > Nothing else. This is needed because booting of EFI labeled disks is > not spuurted (yet).That is what the current 2008.11 dev builds do.>> But what do I do with that swap slice? Should I ditch it and create >> an rpool/swap area? Do I still need a boot slice?Depending on where it is in the VTOC you maybe able to consolidate it into the pool. Only do this if it as at the END of the area used for the pool and DON''t add it as a new vdev let format(1M) grow the slice. As for the boot slice you can''t actually do anything about that and it is unique to x86 (SPARC doesn''t have s8 and s9). -- Darren J Moffat
Hi, maybe this [1] will help you. For more Information read also the linked Blog [2]. HTH [1] http://darkstar-solaris.blogspot.com/2008/09/zfs-root-mirror.html [2] http://malsserver.blogspot.com/2008/08/mirroring-resolved-correct-way.html -- This message posted from opensolaris.org
On Wed, 26 Nov 2008 15:29:50 +0100 "C. Bergstr?m" <cbergstrom at netsyncro.com> wrote:> To clear up some confusion.. > This is from a default indiana install > format -e > verify.. > Part Tag Flag Cylinders Size Blocks > 0 root wm 262 - 19453 147.02GB (19192/0/0) > 308319480 1 swap wu 1 - 261 2.00GB > (261/0/0) 4192965 > > So clearly root is on 0 and swap is on 1.. You *can not* install zfs > to a whole disk and expect it to boot..How can you say this? SXCE and S10U6 are both installed on a disk with only a S0 ZFS takes care of the swap space! I do not only expect it to boot; I see it boot whenever I want to ;-) Current partition table (original): [SXCE.b103] Total disk cylinders available: 19454 + 2 (reserved cylinders) Part Tag Flag Cylinders Size 0 root wm 3 - 19453 149.00GB 1 unassigned wu 0 0 2 backup wm 0 - 19453 149.03GB Current partition table (original): [S10U6] Total disk cylinders available: 38910 + 2 (reserved cylinders) Part Tag Flag Cylinders Size 0 unassigned wm 3 - 38909 298.04GB 1 unassigned wm 0 0 2 backup wu 0 - 38909 298.07GB -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv101 ++ + All that''s really worth doing is what we do for others (Lewis Carrol)
On Wed, 26 Nov 2008 14:37:21 +0000 Darren J Moffat <darrenm at opensolaris.org> wrote:> dick hoogendijk wrote:> > I''ve never seen a ZFS system on seperate slices. Slices are things > > from the past ;-) > > Unfortunately not the case for ZFS pools that are to be booted from. > This is because we can''t boot from an EFI labelled disk.That''s exactly what I wrote, didn''t I. I mentioned SEPERATE slices. I know we still need ONE slice if we want to boot off the disk ;-) -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv101 ++ + All that''s really worth doing is what we do for others (Lewis Carrol)
Darren J Moffat wrote:> dick hoogendijk wrote: > >> On Wed, 26 Nov 2008 12:51:04 +0000 >> Chris Ridd <chrisridd at mac.com> wrote: >> >> >>> I''m replacing the disk with my rpool with a mirrored pool, and >>> wondering how best to do that. >>> >>> The disk I''m replacing is partitioned with root on s0, swap on s1 >>> and boot on s8, which is what the original 2008.05 installer created >>> for me. >>> >> Are you sure about this? OS2008-05 uses the whole disk as a ZFS and >> within that rpool creates the seperate filesystems (swap/dump,..) >> >> > The original builds of OpenSolaris 2008.05 did NOT use the whole disk. > They created a separate swap slice in the Solaris VTOC in side the > SOLARIS2 fdisk partition and put the pool on the rest. That swap slice > was configured as swap and dump. > > >> I''ve never seen a ZFS system on seperate slices. Slices are things from >> the past ;-) >> > > Unfortunately not the case for ZFS pools that are to be booted from. > This is because we can''t boot from an EFI labelled disk. Well maybe > MacOS X hardware could but general x86 systems are still legacy BIOS and > not EFI loaders. > > Even the current 2008.11 development builds still use a Solaris VTOC it > is just that they make the slice the full size of the SOLARIS2 fdisk > partition so it looks like it is using the whole disk. > > >>> to mirror the root. The -f is to stop zpool whining about s0 >>> overlapping s2. >>> >> If I use a disk for a root pool I create just one slice on it (s0). >> Nothing else. This is needed because booting of EFI labeled disks is >> not spuurted (yet). >> > > That is what the current 2008.11 dev builds do. >Ok. here''s a trick question.. So to the best of my understanding zfs turns off write caching if it doesn''t own the whole disk.. So what if s0 *is* the whole disk? Is write cache supposed to be turned on or off? (Haven''t check this locally) Also is it more efficient/better performing to give swap a 2nd slice on the inner part of the disk or not care and just toss it on top of zfs? ./C
dick hoogendijk wrote:> On Wed, 26 Nov 2008 15:29:50 +0100 > "C. Bergstr?m" <cbergstrom at netsyncro.com> wrote: > > >> To clear up some confusion.. >> This is from a default indiana install >> format -e >> verify.. >> Part Tag Flag Cylinders Size Blocks >> 0 root wm 262 - 19453 147.02GB (19192/0/0) >> 308319480 1 swap wu 1 - 261 2.00GB >> (261/0/0) 4192965 >> >> So clearly root is on 0 and swap is on 1.. You *can not* install zfs >> to a whole disk and expect it to boot.. >> > > How can you say this? > SXCE and S10U6 are both installed on a disk with only a S0 > ZFS takes care of the swap space! > I do not only expect it to boot; I see it boot whenever I want to ;-) > >I was directly responding to " I''ve never seen a ZFS system on seperate slices. Slices are things from the past ;-) " slice(s) are still around whether it takes up the whole disk or not.. Also the work-around to the bug which requires -f can be worked around by changing swap to sector 1 afaik.
On 26-Nov-08, at 10:30 AM, C. Bergstr?m wrote:> ... Also is it more efficient/better > performing to give swap a 2nd slice on the inner part of the disk > or not > care and just toss it on top of zfs?I think the thing about swap is that if you''re swapping, you probably have more to worry about than any effect of its physical cylinder. --Toby> > > ./C > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On 26 Nov 2008, at 14:37, Darren J Moffat wrote:> dick hoogendijk wrote: >> On Wed, 26 Nov 2008 12:51:04 +0000 >> Chris Ridd <chrisridd at mac.com> wrote: >>> But what do I do with that swap slice? Should I ditch it and create >>> an rpool/swap area? Do I still need a boot slice? > > > Depending on where it is in the VTOC you maybe able to consolidate it > into the pool. Only do this if it as at the END of the area used for > the pool and DON''t add it as a new vdev let format(1M) grow the slice.Rather inconveniently swap starts at cylinder 1, and the pool starts at 262. So no joy there.> As for the boot slice you can''t actually do anything about that and it > is unique to x86 (SPARC doesn''t have s8 and s9).It feels a lot like "don''t start from here" (ie from my 2008.05 install) so I''m doing an install of 101b from CD onto one of the new disks right now. At least format''s not showing a swap slice now, yippee. I''ll try and get it mirroring afterwards, and then see how much I can extract from my broken disk. Thanks for the clarifications! Cheers, Chris
On Wed, Nov 26, 2008 at 04:30:59PM +0100, "C. Bergstr?m" wrote:> Ok. here''s a trick question.. So to the best of my understanding zfs > turns off write caching if it doesn''t own the whole disk.. So what if s0 > *is* the whole disk? Is write cache supposed to be turned on or off?Actually, ZFS doesn''t turn on write caching if it doesn''t own the whole disk. It leaves it alone. You can turn it on yourself. It leaves it alone because it doesn''t know if it would be safe to enable. If you know that there is nothing on the disk other than ZFS, you can enable it and ZFS will get the benefit.> (Haven''t check this locally) Also is it more efficient/better > performing to give swap a 2nd slice on the inner part of the disk or not > care and just toss it on top of zfs?Swapfile performance is usually pretty far down on the list of things I want to optimize. I''d rather set up for good management and expect that I won''t need high-performance swapfiles. -- Darren
Casper.Dik at Sun.COM
2008-Nov-27 09:42 UTC
[zfs-discuss] Best practice for swap and root pool
>On Wed, Nov 26, 2008 at 04:30:59PM +0100, "C. Bergstr?m" wrote: >> Ok. here''s a trick question.. So to the best of my understanding zfs >> turns off write caching if it doesn''t own the whole disk.. So what if s0 >> *is* the whole disk? Is write cache supposed to be turned on or off? > >Actually, ZFS doesn''t turn on write caching if it doesn''t own the whole >disk. It leaves it alone. You can turn it on yourself. > >It leaves it alone because it doesn''t know if it would be safe to >enable. If you know that there is nothing on the disk other than ZFS, >you can enable it and ZFS will get the benefit.Isn''t it true that for IDE disks, the disks have "write caching" enabled by default?>Swapfile performance is usually pretty far down on the list of things I >want to optimize. I''d rather set up for good management and expect that >I won''t need high-performance swapfiles.If swap performance is a bottleneck, add more memory. Casper
On 26 Nov 2008, at 17:08, Chris Ridd wrote:> It feels a lot like "don''t start from here" (ie from my 2008.05 > install) so I''m doing an install of 101b from CD onto one of the new > disks right now. At least format''s not showing a swap slice now, > yippee. I''ll try and get it mirroring afterwards, and then see how > much I can extract from my broken disk.I''ve successfully got 101b installed and booting and mirrored. I did get a "Bad PBR sig" error after the install onto a single disk, and subsequently noticed the slices on that disk were slightly off: slice 8 (boot) was on cylinder 0, but slices 0 and 2 ended on different cylinders. I adjusted the end of slice 0, ran installgrub on it, and it now boots. I copied that vtoc over to the other disk and ran installgrub on it too. I''m not 100% convinced it''ll boot if half the mirror''s not there, but I was rearranging the drives on the controller at the time. Presumably that''s a terrible idea? Now to recover the bits from my dying disk... Cheers, Chris
On Thu, 27 Nov 2008 12:58:20 +0000 Chris Ridd <chrisridd at mac.com> wrote:> I''m not 100% convinced it''ll boot if half the mirror''s not there,Believe me, it will (been there done that). You -have- to make sure though that both disks have installgrub ... And that your bios is able to boot from the other disk. You can always try it ou by pulling a plug from one of the disks ;-) -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv101 ++ + All that''s really worth doing is what we do for others (Lewis Carrol)
Peter Brouwer, Principal Storage Architect
2008-Nov-28 00:07 UTC
[zfs-discuss] Best practice for swap and root pool
An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081128/ca3823d5/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3303 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081128/ca3823d5/attachment.bin>
On 28 Nov 2008, at 00:07, Peter Brouwer, Principal Storage Architect wrote:> > > dick hoogendijk wrote: >> >> On Thu, 27 Nov 2008 12:58:20 +0000 >> Chris Ridd <chrisridd at mac.com> wrote: >> >> >>> I''m not 100% convinced it''ll boot if half the mirror''s not there, >>> >> Believe me, it will (been there done that). You -have- to make sure >> though that both disks have installgrub ... And that your bios is >> able >> to boot from the other disk.Nod. I''ve had one boot failure already, but maybe that was down to the mirrors rebuilding while trying to boot? The result was an error in grub: "Error 37: file system not found" (IIRC). Google doesn''t tell me much about that one. I was able to boot from the live CD and import the rpool OK, and rebooting off the disks then succeeded. As long as I can do that, I''m sort of happy.>> You can always try it ou by pulling a plug from one of the disks ;-) >> > Or less drastic, use the BIOS to point to one of the disks in the > mirror to boot from. > If you have used installgrub to setup the stage 1&2 for both boot > drives you can boot from either one.That wouldn''t test if there was some interdependency between the disks - I''m not sure how such a thing would happen but with my luck this week I''m sure I could do it. Is there a better list to discuss boot problems? Cheers, Chris