zfs discuss - Jan 2008 - Replacing Devices in a Storage Pool

Hi,

Assume that you have a 2-way mirror of small drives that you want to replace
with another 2-way mirror of larger drives. What is the best way to do this?

If you use the zpool replace command, one at a time on each of the existing old
drives, then you will end up wasting the additional space on the new drives. I
have tried to access this space by creating a new pool and adding the unused
space, but it does not show up as a slice. I also checked the partition table
and the whole disk appears to be used.

How do you get around this? Should I initially have created slices on the bigger
disks and then mirrored the slices (same size as the smaller disks)? Or is there
some way to migrate data from one mirror (or pool) to another?
 
 
This message posted from opensolaris.org

Hi Kava,

Your questions are hard for me to answer without seeing your syntax.

Also, you don''t need to futz with slices if you are using whole disks.

I added some add''l information to the zpool replace section
on page 74, here:

http://opensolaris.org/os/community/zfs/docs/zfsadmin.pdf

Note that an existing bug prevents the expanded disk space
from a zpool replace operation to be seen without doing
an zpool export and zpool import of the pool.

Cindy




Kava wrote:
> Hi,
> 
> Assume that you have a 2-way mirror of small drives that you want to
replace with another 2-way mirror of larger drives. What is the best way to do
this?
> 
> If you use the zpool replace command, one at a time on each of the existing
old drives, then you will end up wasting the additional space on the new drives.
I have tried to access this space by creating a new pool and adding the unused
space, but it does not show up as a slice. I also checked the partition table
and the whole disk appears to be used.
> 
> How do you get around this? Should I initially have created slices on the
bigger disks and then mirrored the slices (same size as the smaller disks)? Or
is there some way to migrate data from one mirror (or pool) to another?
>  
>  
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

You do not waste the new space

zfs seems to always work of the lowest device - so when you replace the disks
with larger ones it increases the mirror size to these new drives as thats the
new smallest drive.
 
 
This message posted from opensolaris.org

I don''t think that is correct. I did it 5 minutes ago and it
didn''t change the pool size at all.

Here is what I did:
- create mirrored pool of 2 x 8GB disks 
- detach one disk
- attach/replace with 12Gb disk
- detach second disk
- attach/replace with second 12GB disk

After this, the pool was still 8GB in size and there was no
''spare'' space that I could see or use on the 12GB disks.

Maybe I did it wrong?
 
 
This message posted from opensolaris.org

Kava,

Because of a recent bug, you need to export and import the pool to see 
the expanded space after you use zpool replace.

Also, you don''t need to detach first. The process would look like this:

# zpool create test mirror 8gb-1 8gb-2
# zpool replace test 8gb-1 12gb-1
# zpool replace test 8gb-2 12gb-2
# zpool export test
# zpool import test

See if that helps...

Cindy


Kava wrote:
> I don''t think that is correct. I did it 5 minutes ago and it
didn''t change the pool size at all.
> 
> Here is what I did:
> - create mirrored pool of 2 x 8GB disks 
> - detach one disk
> - attach/replace with 12Gb disk
> - detach second disk
> - attach/replace with second 12GB disk
> 
> After this, the pool was still 8GB in size and there was no
''spare'' space that I could see or use on the 12GB disks.
> 
> Maybe I did it wrong?
>  
>  
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

I finally got this to work, but it did not happen automatically. I needed to
export then re-import the pool to get it to work. Only then did the additional
space appear.

Here is what I did:
- create 4 x 8GB disks and 1 x 4 GB disks
- create RAIDZ pool with 3 x 8GB disks & 1 x 4GB 
- ignore warning about wasted space
- end up with pool of 12GB usable (4 x 4GB less 4GB parity)
- replace 4GB disk with 8GB disk
- check pool stats and note that it still has 12GB usable space
- export pool (use defaults)
- import pool (use defaults)
- check pool stats and note that usable space is now 24GB
 
 
This message posted from opensolaris.org

> Your questions are hard for me to answer without seeing your syntax.
> 
> Also, you don''t need to futz with slices if you are using whole
disks.
I think what he''s asking is if it''s possible to replace a
whole mirror or
RAID-Z vdev in one go. For instance, replacing a small mirror with a bigger
one requires replacing each disk of the existing mirror one after another
sequentially.

The most ideal solution would be a replace in a single go, which would
however require the ability to address these vdevs. Example how it should
be:

zpool replace tank/0xDEADBEEF mirror c1t0d0 c1t0d

Where the 0xDEADBEEF part would be a random short ID assigned to a
mirror/RAID-Z vdev to let you target it specifically (not possible right
now, AFAIK). Addressability would also help with removals in future.

Regards,
-mg

Thanks .. I stumbled on the export/import as a solution by acciden (see post
below). What confuses me though is that a number of people are reporting that
they do not need to do this step.

See JMCP and others in this thread:
http://www.opensolaris.org/jive/thread.jspa?threadID=50052&tstart=0

I am using the pre-built VM, so I suppose it is possible that they have a
slightly different version?
 
 
This message posted from opensolaris.org