zfs discuss - Dec 2008 - Easiest way to replace a boot disk with a larger one?

Maybe this has been discussed before, but I haven''t been able to find
any relevant threads.

I have a simple OpenSolaris 2008.11 setup with one ZFS pool consisting of the
whole of the single hard drive on the system. What I want to do is to replace
the present 500 GB drive with a 1.5 TB drive. (The latter costs what the former
cost a year ago. :-) Once the replacement is complete, I will install a second
1.5 TB drive to mirror the first one. The smaller drive will go into my legacy
Linux box.)

The way I hope I can do this is by first using the larger drive to mirror the
smaller one. Once the silvering is complete, I would remove the smaller drive.
My question is: once the smaller drive has been removed, will the zpool use all
of the larger, replacement drive?

The ZFS Administration Guide does not appear to give an answer to this. The only
thing I could find in the December 2008 version is the following about
"Replacing Devices in a Storage Pool" on p. 115: "If the
replacement device is larger, the pool capacity is increased when the
replacement is complete." But "zpool replace" does not seem
relevant to what I want to do, since I don''t see how you can use the
procedure described there to replace a drive which comprises the root zpool,
which is what I want to do.

The only thing I have been able to find about this anywhere is the following
from the Wikipedia article on ZFS:

"Capacity expansion is normally achieved by adding groups of disks as a
vdev (stripe, RAID-Z, RAID-Z2, or mirrored). Newly written data will dynamically
start to use all available vdevs. It is also possible to expand the array by
iteratively swapping each drive in the array with a bigger drive and waiting for
ZFS to heal itself ? the heal time will depend on amount of stored information,
not the disk size. The new free space will not be available until all the disks
have been swapped."

Is this correct?
-- 
This message posted from opensolaris.org

On Thu, Dec 11, 2008 at 12:43 PM, Alex Viskovatoff <aufgehoben at
imap.cc>wrote:
> Maybe this has been discussed before, but I haven''t been able to
find any
> relevant threads.
>
> I have a simple OpenSolaris 2008.11 setup with one ZFS pool consisting of
> the whole of the single hard drive on the system. What I want to do is to
> replace the present 500 GB drive with a 1.5 TB drive. (The latter costs
what
> the former cost a year ago. :-) Once the replacement is complete, I will
> install a second 1.5 TB drive to mirror the first one. The smaller drive
> will go into my legacy Linux box.)
>
> The way I hope I can do this is by first using the larger drive to mirror
> the smaller one. Once the silvering is complete, I would remove the smaller
> drive. My question is: once the smaller drive has been removed, will the
> zpool use all of the larger, replacement drive?
>
> The ZFS Administration Guide does not appear to give an answer to this. The
> only thing I could find in the December 2008 version is the following about
> "Replacing Devices in a Storage Pool" on p. 115: "If the
replacement device
> is larger, the pool capacity is increased when the replacement is
complete."
> But "zpool replace" does not seem relevant to what I want to do,
since I
> don''t see how you can use the procedure described there to replace
a drive
> which comprises the root zpool, which is what I want to do.
>
> The only thing I have been able to find about this anywhere is the
> following from the Wikipedia article on ZFS:
>
> "Capacity expansion is normally achieved by adding groups of disks as
a
> vdev (stripe, RAID-Z, RAID-Z2, or mirrored). Newly written data will
> dynamically start to use all available vdevs. It is also possible to expand
> the array by iteratively swapping each drive in the array with a bigger
> drive and waiting for ZFS to heal itself ? the heal time will depend on
> amount of stored information, not the disk size. The new free space will
not
> be available until all the disks have been swapped."
>
> Is this correct?
>


As far as I know it should be as simple as (where newdisk and olddisk are
the actual cXtXdX of your drives):
zpool attach rpool olddisk-s0 newdisk-s0

let it completely resilver then:
installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/newdisks0
zpool detach rpool oldisk-s0

reboot and you should now see the new space.


--Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081211/bac0ce65/attachment.html>

Thanks, that''s what I thought. Just wanted to make sure.

I guess the writers of the documentation think that this is so obviously the way
things would work in a well designed system that there is no reason to mention
it explicitly.
-- 
This message posted from opensolaris.org

Hi Alex,

Not exactly. Just hadn''t thought of that specific example yet, but its
a
good one so I''ll add it.

In your case, ZFS might not see the expanded capacity of the larger disk 
automatically due to a recent bug. For non-root pools, the workaround to 
see the expanded space is to export and import the pool, but you can''t
export an active root pool. So, as Tim says, reboot after the disk is
replaced and the bootblocks are installed.

Ask Santa for another disk, so you can create a mirrored root pool.

Cindy

Alex Viskovatoff wrote:
> Thanks, that''s what I thought. Just wanted to make sure.
> 
> I guess the writers of the documentation think that this is so obviously
the way things would work in a well designed system that there is no reason to
mention it explicitly.

Hi Cindy,

Thanks for clearing that up. I don''t mind rebooting, just as long as
that makes the zpool use the additional space. I did read about the
export/import workaround, but wasn''t sure if rebooting would have the
same effect.

The ZFS documentation convinced me to set up a mirrored pool, even though
I''ve never mirrored disks or used RAID before. So I''ve already
put in a word with Santa about a second disk.

Btw, I would never consider using a disk with bleeding-edge capacity for my
system (as opposed to for expendable data like movies) with any file system
other than ZFS.

Alex
-- 
This message posted from opensolaris.org

Hi again Cindy,

Well, I got the two new 1.5 TB disks, but I ran into a snag:
> av at diotima:~# zpool attach rpool c3t0d0s0 c3t1d0
> cannot label ''c3t1d0'': EFI labeled devices are not
supported on root pools.
The Solaris 10 System Administration Guide: Devices and File Systems gives some
pertinent information on p. 191.
> Comparison of the EFI Label and the VTOC Label
> The EFI disk label differs from the VTOC disk label in the following ways:
>    Provides support for disks greater than 1 terabyte in size.
Well, that explains why my new disks have EFI labels; furthermore, it appears to
mean that I must keep the EFI label if I want to use the full capacity of the
disks.
> Restrictions of the EFI Disk Label
> Keep the following restrictions in mind when determining whether using
disks greater than 1
   terabyte is appropriate for your environment:
>     [...]
>      You cannot boot from a disk with an EFI disk label.
It thus appears that my original plan to replace my 500 GB boot disk with a 1.5
TB boot disk was ill advised, since there is at present no such thing as a 1.5
TB Solaris boot disk.

I was thinking of trying an install of OS 2008.11 one of the 1.5 TB disks, just
to see what would happen, but then I found the following thread:
http://opensolaris.org/jive/thread.jspa?threadID=81852&tstart=0

which refers to this "bug":
> 4051 opensolaris b99b/b100a does not install on 1.5 TB disk or boot fails
after install
It thus appears that I will have to create a new non-root zpool for the 1.5 TB
disks. (Fortunately, I have an extra 500 GB disk which I?m now going to set up
to mirror my boot disk.)

Does anyone know if there are any plans in the work to make disks with an EFI
disk label bootable? The present situation of their not being bootable places
the restriction on OpenSolaris that it cannot boot off disks greater than 1 TB.
-- 
This message posted from opensolaris.org

Hi Alex,

Sorry, I missed the 1.5 TB disk/boot issue previously.

A project is underway to provide booting for disks that are large
than 1 TB. This project is outside of a future project to provide
booting from an EFI-labeled disk.

I agree that your best approach is to attach your second 500-GB disk
to create a mirrored root pool and save the 1.5 TB disk for another
day or another pool. You will be able to replace the 500-GB mirrored
root pool with the 1.5 TB when booting from it is supported.

What I''m thinking is that you will be able to upgrade your root pool
with a release that supports booting from a 1.5 TB disk, then do
the disk replacement.

Cindy


Alex Viskovatoff wrote:
> Hi again Cindy,
> 
> Well, I got the two new 1.5 TB disks, but I ran into a snag:
> 
> 
>>av at diotima:~# zpool attach rpool c3t0d0s0 c3t1d0
>>cannot label ''c3t1d0'': EFI labeled devices are not
supported on root pools.
> 
> 
> The Solaris 10 System Administration Guide: Devices and File Systems gives
some pertinent information on p. 191.
> 
> 
>>Comparison of the EFI Label and the VTOC Label
>>The EFI disk label differs from the VTOC disk label in the following
ways:
>>   Provides support for disks greater than 1 terabyte in size.
> 
> 
> Well, that explains why my new disks have EFI labels; furthermore, it
appears to mean that I must keep the EFI label if I want to use the full
capacity of the disks.
> 
> 
>>Restrictions of the EFI Disk Label
>>Keep the following restrictions in mind when determining whether using
disks greater than 1
> 
>    terabyte is appropriate for your environment:
> 
>>    [...]
>>     You cannot boot from a disk with an EFI disk label.
> 
> 
> It thus appears that my original plan to replace my 500 GB boot disk with a
1.5 TB boot disk was ill advised, since there is at present no such thing as a
1.5 TB Solaris boot disk.
> 
> I was thinking of trying an install of OS 2008.11 one of the 1.5 TB disks,
just to see what would happen, but then I found the following thread:
http://opensolaris.org/jive/thread.jspa?threadID=81852&tstart=0
> 
> which refers to this "bug":
> 
> 
>>4051 opensolaris b99b/b100a does not install on 1.5 TB disk or boot
fails after install
> 
> 
> It thus appears that I will have to create a new non-root zpool for the 1.5
TB disks. (Fortunately, I have an extra 500 GB disk which I?m now going to set
up to mirror my boot disk.)
> 
> Does anyone know if there are any plans in the work to make disks with an
EFI disk label bootable? The present situation of their not being bootable
places the restriction on OpenSolaris that it cannot boot off disks greater than
1 TB.

On Wed, Dec 17, 2008 at 10:23 AM, <Cindy.Swearingen at sun.com> wrote:
> Hi Alex,
>
> Sorry, I missed the 1.5 TB disk/boot issue previously.
>
> A project is underway to provide booting for disks that are large
> than 1 TB. This project is outside of a future project to provide
> booting from an EFI-labeled disk.
>
> I agree that your best approach is to attach your second 500-GB disk
> to create a mirrored root pool and save the 1.5 TB disk for another
> day or another pool. You will be able to replace the 500-GB mirrored
> root pool with the 1.5 TB when booting from it is supported.
>
> What I''m thinking is that you will be able to upgrade your root
pool
> with a release that supports booting from a 1.5 TB disk, then do
> the disk replacement.
>
> Cindy
>
>
Since I''ve heard differing stories over time: Is the EFI boot
limitation a
limitation of PC BIOS, or is it somewhere in the software stack?  Or I
suppose the third options is that it''s a limitation of PC BIOS that can
be
overcome with some software hacks.
>From what I recall, itanium systems and apple systems have no issues booting
from an EFI label.  I would assume this may apply to sparc based systems as
well since they''re also running OBP (like apple).

Thoughts?

--Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081217/7456bd78/attachment.html>

On Wed, Dec 17, 2008 at 01:57:37PM -0600, Tim wrote:
> On Wed, Dec 17, 2008 at 10:23 AM, <Cindy.Swearingen at sun.com>
wrote:
> 
> > Hi Alex,
> >
> > Sorry, I missed the 1.5 TB disk/boot issue previously.
> >
> > A project is underway to provide booting for disks that are large
> > than 1 TB. This project is outside of a future project to provide
> > booting from an EFI-labeled disk.
> Since I''ve heard differing stories over time: Is the EFI boot
limitation a
> limitation of PC BIOS, or is it somewhere in the software stack?  Or I
> suppose the third options is that it''s a limitation of PC BIOS
that can be
> overcome with some software hacks.
I belive both would be trouble.  I suppose if you could put both labels
on the disk you could have a hybrid that allows MBR booting but could
allow EFI access later.  But that gets tricky (if it''s possible).

However the 1TB limitation is on the Solaris VTOC, not on the MBR
label.  It''s possible that you could make some modifications to Solaris
to allow a (sub-1TB) MBR partition holding a Solaris VTOC label for
booting and then use the rest of the disk for ZFS.

That''s how I read the previous message.
> >From what I recall, itanium systems and apple systems have no issues
booting
> from an EFI label.
Right.  EFI was part of the Itanium stuff, and Apple supported it in
their hardware as well.  But standard BIOS doesn''t.
> I would assume this may apply to sparc based systems as
> well since they''re also running OBP (like apple).
Pre-EFI apple hardware ran OpenFirmware.  While I don''t think it would
be impossible to add EFI/GPT label support to it, I doubt there''s much
clamor for it.  As far as I am aware, Apple''s EFI machines do not use
OpenFirmware any longer.

-- 
Darren

>>>>> "t" == Tim  <tim at tcsac.net> writes:
>>>>> "add" == A Darren Dunham <ddunham at
taos.com> writes:
     t> Is the EFI boot limitation a limitation of PC BIOS,

   add> EFI was part of the Itanium stuff, and Apple supported
   add> it in their hardware as well.  But standard BIOS doesn''t.

I''ve heard (on another list, last week) that CentOS and Debian both
boot off GPT-labeled disks.  I''m not sure if they can do it on ~all
systems or only the newest ones, but I know they are doing it right
now on commodity hardware, not apple not itanium.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 304 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081227/d4ce13c1/attachment.bin>