thr3ads.net - freebsd stable - Where's the space? raidz2 [Aug 2010]

If this information is useful, please help other people find it:
Share via:

Dan Langille

2010-Aug-02 23:12 UTC

Where's the space? raidz2

I recently altered an existing raidz2 pool from using 7 vdevs of about 
931G to 1.81TB.  In fact, the existing pool used half of each HDD.  I 
then wanted to go to using [almost] all of each HDD.

I offline'd each vdev, adjusted the HDD paritions using gpart, then 
replaced the vdev.  After letting the resilver occur, I did the next vdev.

The space available after this process did not go up as I expected.  I 
have about 4TB in the pool, not the 8 or 9TB I expected.

At present, I have this:

$ zpool list
NAME      SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
storage  6.31T  5.51T   819G    87%  ONLINE  -

Which is:

$ zpool status
   pool: storage
  state: ONLINE
  scrub: scrub in progress for 3h26m, 63.33% done, 1h59m to go
config:

         NAME                 STATE     READ WRITE CKSUM
         storage              ONLINE       0     0     0
           raidz2             ONLINE       0     0     0
             gpt/disk01-live  ONLINE       0     0     0
             gpt/disk02-live  ONLINE       0     0     0
             gpt/disk03-live  ONLINE       0     0     0
             gpt/disk04-live  ONLINE       0     0     0
             gpt/disk05-live  ONLINE       0     0     0
             gpt/disk06-live  ONLINE       0     0     0
             gpt/disk07-live  ONLINE       0     0     0

errors: No known data errors


The disks involved are:

$ gpart show ada0 ada1 ada2 ada3 ada4 ada5 ada6
=>        34  3907029101  ada0  GPT  (1.8T)
           34        2014        - free -  (1.0M)
         2048  3906617453     1  freebsd-zfs  (1.8T)
   3906619501      409634        - free -  (200M)

=>        34  3907029101  ada1  GPT  (1.8T)
           34        2014        - free -  (1.0M)
         2048  3906617453     1  freebsd-zfs  (1.8T)
   3906619501      409634        - free -  (200M)

=>        34  3907029101  ada2  GPT  (1.8T)
           34        2014        - free -  (1.0M)
         2048  3906617453     1  freebsd-zfs  (1.8T)
   3906619501      409634        - free -  (200M)

=>        34  3907029101  ada3  GPT  (1.8T)
           34        2014        - free -  (1.0M)
         2048  3906617453     1  freebsd-zfs  (1.8T)
   3906619501      409634        - free -  (200M)

=>        34  3907029101  ada4  GPT  (1.8T)
           34        2014        - free -  (1.0M)
         2048  3906617453     1  freebsd-zfs  (1.8T)
   3906619501      409634        - free -  (200M)

=>        34  3907029101  ada5  GPT  (1.8T)
           34        2014        - free -  (1.0M)
         2048  3906617453     1  freebsd-zfs  (1.8T)
   3906619501      409634        - free -  (200M)

=>        34  3907029101  ada6  GPT  (1.8T)
           34        2014        - free -  (1.0M)
         2048  3906617453     1  freebsd-zfs  (1.8T)
   3906619501      409634        - free -  (200M)

Checking the sizes are all equal:

$ gpart show ada0 ada1 ada2 ada3 ada4 ada5 ada6 | grep freebsd-zfs
         2048  3906617453     1  freebsd-zfs  (1.8T)
         2048  3906617453     1  freebsd-zfs  (1.8T)
         2048  3906617453     1  freebsd-zfs  (1.8T)
         2048  3906617453     1  freebsd-zfs  (1.8T)
         2048  3906617453     1  freebsd-zfs  (1.8T)
         2048  3906617453     1  freebsd-zfs  (1.8T)
         2048  3906617453     1  freebsd-zfs  (1.8T)


The labels on these are:

$ gpart list ada0 ada1 ada2 ada3 ada4 ada5 ada6  | grep label
    label: disk06-live
    label: disk01-live
    label: disk02-live
    label: disk03-live
    label: disk04-live
    label: disk05-live
    label: disk07-live


So, I'm not sure where I went wrong yet...  far more information than 
you ever wanted to know follows:

$ gpart list
Geom name: ada2
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
    Mediasize: 2000188135936 (1.8T)
    Sectorsize: 512
    Mode: r1w1e2
    rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
    label: disk02-live
    length: 2000188135936
    offset: 1048576
    type: freebsd-zfs
    index: 1
    end: 3906619500
    start: 2048
Consumers:
1. Name: ada2
    Mediasize: 2000398934016 (1.8T)
    Sectorsize: 512
    Mode: r1w1e3

Geom name: mirror/gm0
fwheads: 255
fwsectors: 63
last: 156301424
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: mirror/gm0s1
    Mediasize: 80026329600 (75G)
    Sectorsize: 512
    Mode: r5w5e9
    attrib: active
    rawtype: 165
    length: 80026329600
    offset: 32256
    type: freebsd
    index: 1
    end: 156301487
    start: 63
Consumers:
1. Name: mirror/gm0
    Mediasize: 80026361344 (75G)
    Sectorsize: 512
    Mode: r5w5e14

Geom name: mirror/gm0s1
fwheads: 255
fwsectors: 63
last: 156301424
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mirror/gm0s1a
    Mediasize: 1073741824 (1.0G)
    Sectorsize: 512
    Mode: r1w1e1
    rawtype: 7
    length: 1073741824
    offset: 0
    type: freebsd-ufs
    index: 1
    end: 2097151
    start: 0
2. Name: mirror/gm0s1b
    Mediasize: 6442450944 (6.0G)
    Sectorsize: 512
    Mode: r1w1e0
    rawtype: 1
    length: 6442450944
    offset: 1073741824
    type: freebsd-swap
    index: 2
    end: 14680063
    start: 2097152
3. Name: mirror/gm0s1d
    Mediasize: 4294967296 (4.0G)
    Sectorsize: 512
    Mode: r1w1e1
    rawtype: 7
    length: 4294967296
    offset: 7516192768
    type: freebsd-ufs
    index: 4
    end: 23068671
    start: 14680064
4. Name: mirror/gm0s1e
    Mediasize: 4294967296 (4.0G)
    Sectorsize: 512
    Mode: r1w1e1
    rawtype: 7
    length: 4294967296
    offset: 11811160064
    type: freebsd-ufs
    index: 5
    end: 31457279
    start: 23068672
5. Name: mirror/gm0s1f
    Mediasize: 63920202240 (60G)
    Sectorsize: 512
    Mode: r1w1e1
    rawtype: 7
    length: 63920202240
    offset: 16106127360
    type: freebsd-ufs
    index: 6
    end: 156301424
    start: 31457280
Consumers:
1. Name: mirror/gm0s1
    Mediasize: 80026329600 (75G)
    Sectorsize: 512
    Mode: r5w5e9

Geom name: ada0
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
    Mediasize: 2000188135936 (1.8T)
    Sectorsize: 512
    Mode: r1w1e2
    rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
    label: disk06-live
    length: 2000188135936
    offset: 1048576
    type: freebsd-zfs
    index: 1
    end: 3906619500
    start: 2048
Consumers:
1. Name: ada0
    Mediasize: 2000398934016 (1.8T)
    Sectorsize: 512
    Mode: r1w1e3

Geom name: ada6
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada6p1
    Mediasize: 2000188135936 (1.8T)
    Sectorsize: 512
    Mode: r1w1e2
    rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
    label: disk07-live
    length: 2000188135936
    offset: 1048576
    type: freebsd-zfs
    index: 1
    end: 3906619500
    start: 2048
Consumers:
1. Name: ada6
    Mediasize: 2000398934016 (1.8T)
    Sectorsize: 512
    Mode: r1w1e3

Geom name: ada1
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
    Mediasize: 2000188135936 (1.8T)
    Sectorsize: 512
    Mode: r1w1e2
    rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
    label: disk01-live
    length: 2000188135936
    offset: 1048576
    type: freebsd-zfs
    index: 1
    end: 3906619500
    start: 2048
Consumers:
1. Name: ada1
    Mediasize: 2000398934016 (1.8T)
    Sectorsize: 512
    Mode: r1w1e3

Geom name: ada3
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
    Mediasize: 2000188135936 (1.8T)
    Sectorsize: 512
    Mode: r1w1e2
    rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
    label: disk03-live
    length: 2000188135936
    offset: 1048576
    type: freebsd-zfs
    index: 1
    end: 3906619500
    start: 2048
Consumers:
1. Name: ada3
    Mediasize: 2000398934016 (1.8T)
    Sectorsize: 512
    Mode: r1w1e3

Geom name: ada4
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada4p1
    Mediasize: 2000188135936 (1.8T)
    Sectorsize: 512
    Mode: r1w1e2
    rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
    label: disk04-live
    length: 2000188135936
    offset: 1048576
    type: freebsd-zfs
    index: 1
    end: 3906619500
    start: 2048
Consumers:
1. Name: ada4
    Mediasize: 2000398934016 (1.8T)
    Sectorsize: 512
    Mode: r1w1e3

Geom name: ada5
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada5p1
    Mediasize: 2000188135936 (1.8T)
    Sectorsize: 512
    Mode: r1w1e2
    rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
    label: disk05-live
    length: 2000188135936
    offset: 1048576
    type: freebsd-zfs
    index: 1
    end: 3906619500
    start: 2048
Consumers:
1. Name: ada5
    Mediasize: 2000398934016 (1.8T)
    Sectorsize: 512
    Mode: r1w1e3



-- 
Dan Langille - http://langille.org/

Dan Langille

2010-Aug-02 23:28 UTC

head link

Where's the space? raidz2

On 8/2/2010 7:11 PM, Dan Langille wrote:> I recently altered an existing raidz2 pool from using 7 vdevs of about
> 931G to 1.81TB. In fact, the existing pool used half of each HDD. I then
> wanted to go to using [almost] all of each HDD.
>
> I offline'd each vdev, adjusted the HDD paritions using gpart, then
> replaced the vdev. After letting the resilver occur, I did the next vdev.
>
> The space available after this process did not go up as I expected. I
> have about 4TB in the pool, not the 8 or 9TB I expected.
This fixed it:

# df -h
Filesystem            Size    Used   Avail Capacity  Mounted on
/dev/mirror/gm0s1a    989M    508M    402M    56%    /
devfs                 1.0K    1.0K      0B   100%    /dev
/dev/mirror/gm0s1e    3.9G    500K    3.6G     0%    /tmp
/dev/mirror/gm0s1f     58G    4.6G     48G     9%    /usr
/dev/mirror/gm0s1d    3.9G    156M    3.4G     4%    /var
storage               512G    1.7G    510G     0%    /storage
storage/pgsql         512G    1.7G    510G     0%    /storage/pgsql
storage/bacula        3.7T    3.2T    510G    87%    /storage/bacula
storage/Retored       510G     39K    510G     0%    /storage/Retored


# zpool export storage
# zpool import storage

# df -h
Filesystem            Size    Used   Avail Capacity  Mounted on
/dev/mirror/gm0s1a    989M    508M    402M    56%    /
devfs                 1.0K    1.0K      0B   100%    /dev
/dev/mirror/gm0s1e    3.9G    500K    3.6G     0%    /tmp
/dev/mirror/gm0s1f     58G    4.6G     48G     9%    /usr
/dev/mirror/gm0s1d    3.9G    156M    3.4G     4%    /var
storage               5.0T    1.7G    5.0T     0%    /storage
storage/Retored       5.0T     39K    5.0T     0%    /storage/Retored
storage/bacula        8.2T    3.2T    5.0T    39%    /storage/bacula
storage/pgsql         5.0T    1.7G    5.0T     0%    /storage/pgsql



-- 
Dan Langille - http://langille.org/

freebsd stable - Aug 2010 - Where's the space? raidz2

Where's the space? raidz2

Where's the space? raidz2