Jim Klimov
2012-Nov-11 13:32 UTC
[zfs-discuss] Expanding a ZFS pool disk in Solaris 10 on VMWare (or other expandable storage technology)
Hello all,
This is not so much a question but rather a "how-to" for posterity.
Comments and possible fixes are welcome, though.
I''m toying (for work) with a Solaris 10 VM, and it has a dedicated
virtual HDD for data and zones. The template VM had a 20Gb disk,
but a particular application needs more. I hoped ZFS autoexpand
would do the trick transparently, but it turned out some manual
labor is needed. Even though this was a single-disk pool with a
dedicated disk, the EFI label and ultimate Solaris partition and
slices did not detect the size change over several devfsadm''s,
cfgadm''s, exports, imports, reboots, formats and fdisks - only
when I entered some values by hand did things work out.
Many solutions just added a bigger disk, mirrored the pool and
removed the smaller disk, and were left with a bigger pool.
Sometimes for whatever architectural reason, this way is blocked.
Several blogs and forums I googled up pointed me in the right
direction, so I''d try to overview the solution in one post :)
0) Backup the VM (disk image, SAN LUN, etc.) in question.
Note that depending on the VM technology, disk expansion
may not be available if the VM has snapshots, so you may
have to make backups or snapshots at real storage level.
1) Run "zpool status poolname" to determine component disks, if
you don''t already know those details:
# zpool status
pool: pool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
errors: No known data errors
pool: rpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c1t0d0s0 ONLINE 0 0 0
errors: No known data errors
So, here I''ll work with "pool" which has a dedicated
disk.
I guess a similar technique should be valid for slice-based
pools in MBR partition tables like "rpool" above, corrected
for different partition naming scheme.
2) Resize the HDD image size as is appropriate in your external
storage (VM hypervisor, SAN, backing zvol, etc.) The pool''s
component''s can''t get smaller now, so by resize I mean
increase.
3) Have the OS detect new disk size (use reconfiguration reboot,
devfsadm, and/or cfgadm):
# devfsadm -Cv
and enable autoexpansion (didn''t help me much, hence the post):
# zpool set autoexpand=on pool
4) Run solaris "format" utility and select the needed disk by
number, i.e.:
# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0 <DEFAULT cyl 2607 alt 2 hd 255 sec 63>
/pci at 0,0/pci15ad,7a0 at 15/pci15ad,1976 at 0/sd at 0,0
1. c1t1d0 <VMware-Virtual disk-1.0-60.00GB>
/pci at 0,0/pci15ad,7a0 at 15/pci15ad,1976 at 0/sd at 1,0
Specify disk (enter its number): 1
selecting c1t1d0
[disk formatted]
/dev/dsk/c1t1d0s0 is part of active ZFS pool pool. Please see zpool(1M).
Now, the system already sees the while disk as 60Gb, but when I
inspect the partition table (entering "p", "p"), I only
see the
original 20Gb (~40mln sector) partition size:
format> p
partition> p
Current partition table (original):
Total disk sectors available: 41926621 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 256 19.99GB 41926621
1 unassigned wm 0 0 0
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
8 reserved wm 41926622 8.00MB 41943005
partition> q
If you skim over this text, please do actually print your partition
table and note the ZFS slice (i.e. "0") size and start location in
sectors. Perhaps, write them down on paper.
5) So I go into fdisk to edit lower-level partitions on this drive,
right from the format utility with the disk in question selected:
format> fd
Total disk size is 7832 cylinders
Cylinder size is 16065 (512 byte) blocks
Cylinders
Partition Status Type Start End Length %
========= ====== ============ ===== === ====== == 1
EFI 0 2610 2611 33
Here for a whole disk dedicated to ZFS I see a EFI partition
that does not consume my whole disk. For a root pool''s disk
with MBR I''d see this:
Total disk size is 2610 cylinders
Cylinder size is 16065 (512 byte) blocks
Cylinders
Partition Status Type Start End Length %
========= ====== ============ ===== === ====== == 1
Active Solaris 1 2609 2609 100
If you apply this technique to rpools, note that the partition
type is different (SOLARIS vs EFI), the start cylinders differ
(1 for MBR, 0 for EFI), and the bootable partition is Active.
6) The scary part is that I need to remove the partition and slice
tables and recreate them starting at the same positions.
So in fdisk I press "3" to delete the partition "1",
then I press
"1" to create a new partition. If I select "EFI", it
automatically
fills the disk from 0 to end. An MBR-based (Solaris2) partition
started at 1 and asked me to enter desired size.
For the disk dedicated fully to a pool, I chose "EFI" as it was
originally.
Now I press "5" to save the new partition table and return to
"format". Entering "p","p" I see that the
slice sizes remain as
they were...
Returning to the disk-level menu, I entered "t" for Type:
format> t
AVAILABLE DRIVE TYPES:
0. Auto configure
1. other
Specify disk type (enter its number)[1]: 0
c1t1d0: configured with capacity of 60.00GB
<VMware-Virtual disk-1.0-60.00GB>
selecting c1t1d0
[disk formatted]
/dev/dsk/c1t1d0s0 is part of active ZFS pool pool. Please see zpool(1M).
I picked "0", et voila - the partition sizes are reassigned.
Too early to celebrate however: the ZFS slice #0 now starts at a
wrong position:
format> p
partition> p
Current partition table (default):
Total disk sectors available: 125812701 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 34 59.99GB 125812701
1 unassigned wm 0 0 0
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
8 reserved wm 125812702 8.00MB 125829085
Remembering that the original table started this slice at 256,
and remembering the new table''s last sector value, I mix the two:
partition> 0
Part Tag Flag First Sector Size Last Sector
0 usr wm 34 59.99GB 125812701
Enter partition id tag[usr]:
Enter partition permission flags[wm]:
Enter new starting Sector[34]: 256
Enter partition size[125812668b, 125812923e, 61431mb, 59gb, 0tb]: 125812701e
partition> p
Current partition table (unnamed):
Total disk sectors available: 125812701 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 256 59.99GB 125812701
Finally, I can save the changed tables and exit "format":
partition> label
Ready to label disk, continue? y
partition> q
format> q
7) Inspecting the pool, and even exporting and importing it and
inspecting again, I see that autoexpand did not take place and
the pool is still 20Gb in size (dunno why - sol10u10 bug?) :(
So I do the manual step:
# zpool online -e pool c1t1d0
The "-e" flag marks the component as eligible for expansion.
When all pieces of a top-level vdev become larger, the setting
takes effect and the pool finally becomes larger:
# zpool list
NAME SIZE ALLOC FREE CAP HEALTH ALTROOT
pool 59.9G 441M 59.4G 0% ONLINE -
rpool 19.9G 6.91G 13.0G 34% ONLINE -
Now I can finally go to my primary quest and install that
large piece of software into a zone that lives on "pool"! ;)
HTH,
//Jim Klimov
Possibly Parallel Threads
Wisdom of the Ancients
