zfs discuss - May 2010 - Strategies for expanding storage area of home storage-server

Hello.
I''ve got a home-storage-server setup with Opensolaris (currently dev
build 134) that is quickly running out of storage space, and I''m
looking through what kind of options I have for expanding it.

I currently have my "storage-pool" in a 4x 1TB drive setup in RAIDZ1,
and have room for 8-9 more drives in the case/controllers.
Preferably I''d like to change it all to a RAIDZ2 with 12 drives, and 1
hotspare, but that would require me to transfer out all the data to an external
storage, and then recreating a new pool, which would require me buying some
additional external storage that will not be used after I''m done with
the transfer.

I could also add 2 more 4 drive vdevs to the current pool, but then I would have
3 RAIDZ1 vdevs striped, and I''m not entirely sure that I''m
comfortable with that level of protection on the data.

Another version would be creating a 6 drive RAIDZ2 pool, moving the data to that
one and the destroying the old pool and adding another 6 drive vdev to the new
pool (striped).

So the question is what would you recommend for growing my storage space:
1. Buying extra hardware to copy the data to, and rebuild the pool as a 12 drive
RAIDZ2.
2. Move data to a 6 drive RAIDZ2 and then destroy the old pool and stripe an
additional RAIDZ2 vdevs.
3. Stripe 2 additional RAIDZ1 4 drive vdevs.
4. Something else.

Easiest would of course be adding new 4-drive vdevs to the existing pool, but
I''m unsure how much I''d be able to trust more than 1 drive not
failing in that setup. Am I worried needlessly? (Imagine 10% or so of the data
as vacation-footage or something like that and you''ll be rather close
to how I value the data. I have some backups of the most important stuff, but I
know myself good enough to know I will not backup everything I should as good as
I should). I guess a hotspare on top of the stripe would give some extra buffer
if the drives just resilver fast enough in case of a failure, so that would make
it a bit more "safe".

The hardware setup if anyone''s interested is opensolaris running in a
VM on an  ESXi-server, and storage-pool harddrives are place in a iSCSI target
that I connect to using ESXi''s iSCSI initiator (using MPIO) and
attached to the VM as RAW-devices.

Hope someone has some ideas or opinions.

Regards
Andreas Gunnarsson
-- 
This message posted from opensolaris.org

On Mon, May 17, 2010 at 6:25 AM, Andreas Gunnarsson <andreas at
tiomat.net>wrote:
> I''ve got a home-storage-server setup with Opensolaris (currently
dev build
> 134) that is quickly running out of storage space, and I''m looking
through
> what kind of options I have for expanding it.
>
> I currently have my "storage-pool" in a 4x 1TB drive setup in
RAIDZ1, and
> have room for 8-9 more drives in the case/controllers.
> Preferably I''d like to change it all to a RAIDZ2 with 12 drives,
and 1
> hotspare, but that would require me to transfer out all the data to an
> external storage, and then recreating a new pool, which would require me
> buying some additional external storage that will not be used after
I''m done
> with the transfer.
>
> I could also add 2 more 4 drive vdevs to the current pool, but then I would
> have 3 RAIDZ1 vdevs striped, and I''m not entirely sure that
I''m comfortable
> with that level of protection on the data.
>
> Another version would be creating a 6 drive RAIDZ2 pool, moving the data to
> that one and the destroying the old pool and adding another 6 drive vdev to
> the new pool (striped).
>
> So the question is what would you recommend for growing my storage space:
> 1. Buying extra hardware to copy the data to, and rebuild the pool as a 12
> drive RAIDZ2.
> 2. Move data to a 6 drive RAIDZ2 and then destroy the old pool and stripe
> an additional RAIDZ2 vdevs.
> 3. Stripe 2 additional RAIDZ1 4 drive vdevs.
> 4. Something else.

I''d go with option 2.

Create a 6-drive raidz2 vdev in a separate pool.  Migrate the data from the
old pool to the new pool.  Destroy the old pool.  Create a second 6-drive
raidz2 vdev in the new pool.  Voila!  You''ll have a lot of extra space,
be
able to withstand up to 4 drive failures (2 per vdev), and it should be
faster as well (even with the added overhead of raidz2).

Option 3 would give the best performance, but you don''t have much
leeway in
terms of resilver time if using 1 TB+ drives, and if a second drive fails
while the first is resilvering ...

Option 1 would be horrible in terms of performance.  Especially resilver
times, as you''ll be thrashing 12 drives.

-- 
Freddie Cash
fjwcash at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100517/c1778556/attachment.html>

I''d have to agree.  Option 2 is probably the best.

I recently found myself in need of more space...i had to build an entirely
new server...my first one was close to full (it has 20 1TB drives in 3
raidz2 groups 7/7/6 and i was down to 3 TB)  I ended up going with a whole
new server....with 2TB drives this time...I considered replacing the drives
in my current server with new 2 TB drives but for the money, it made more
sense to keep that server online and build a second....

That''s where i am now...If i could have done what you are looking to
do, it
woudl have been a lot easier....

On Mon, May 17, 2010 at 11:29 AM, Freddie Cash <fjwcash at gmail.com>
wrote:
> On Mon, May 17, 2010 at 6:25 AM, Andreas Gunnarsson <andreas at
tiomat.net>wrote:
>
>> I''ve got a home-storage-server setup with Opensolaris
(currently dev build
>> 134) that is quickly running out of storage space, and I''m
looking through
>> what kind of options I have for expanding it.
>>
>> I currently have my "storage-pool" in a 4x 1TB drive setup in
RAIDZ1, and
>> have room for 8-9 more drives in the case/controllers.
>> Preferably I''d like to change it all to a RAIDZ2 with 12
drives, and 1
>> hotspare, but that would require me to transfer out all the data to an
>> external storage, and then recreating a new pool, which would require
me
>> buying some additional external storage that will not be used after
I''m done
>> with the transfer.
>>
>> I could also add 2 more 4 drive vdevs to the current pool, but then I
>> would have 3 RAIDZ1 vdevs striped, and I''m not entirely sure
that I''m
>> comfortable with that level of protection on the data.
>>
>> Another version would be creating a 6 drive RAIDZ2 pool, moving the
data
>> to that one and the destroying the old pool and adding another 6 drive
vdev
>> to the new pool (striped).
>>
>> So the question is what would you recommend for growing my storage
space:
>> 1. Buying extra hardware to copy the data to, and rebuild the pool as a
12
>> drive RAIDZ2.
>> 2. Move data to a 6 drive RAIDZ2 and then destroy the old pool and
stripe
>> an additional RAIDZ2 vdevs.
>> 3. Stripe 2 additional RAIDZ1 4 drive vdevs.
>> 4. Something else.
>
>
> I''d go with option 2.
>
> Create a 6-drive raidz2 vdev in a separate pool.  Migrate the data from the
> old pool to the new pool.  Destroy the old pool.  Create a second 6-drive
> raidz2 vdev in the new pool.  Voila!  You''ll have a lot of extra
space, be
> able to withstand up to 4 drive failures (2 per vdev), and it should be
> faster as well (even with the added overhead of raidz2).
>
> Option 3 would give the best performance, but you don''t have much
leeway in
> terms of resilver time if using 1 TB+ drives, and if a second drive fails
> while the first is resilvering ...
>
> Option 1 would be horrible in terms of performance.  Especially resilver
> times, as you''ll be thrashing 12 drives.
>
> --
> Freddie Cash
> fjwcash at gmail.com
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100517/0aa0d9fe/attachment.html>

When I did a similar upgrade a while back I did #2. Create a new pool raidz2
with 6 drives, copy the data to it, verify the data, delete the old pool, add
old drives + some new drives to another 6 disk raidz2 in the new pool.
Performance has been quite good, and the migration was very smooth.

The other nice thing about this arrangement for a home user is that I now only
need to upgrade 6 drives to get more space, rather than 12 per option #1. To be
clear, this is my current config.

        NAME         STATE     READ WRITE CKSUM
        raid         ONLINE       0     0     0
          raidz2-0   ONLINE       0     0     0
            c9t4d0   ONLINE       0     0     0
            c9t5d0   ONLINE       0     0     0
            c9t6d0   ONLINE       0     0     0
            c9t7d0   ONLINE       0     0     0
            c10t5d0  ONLINE       0     0     0
            c10t4d0  ONLINE       0     0     0
          raidz2-1   ONLINE       0     0     0
            c9t0d0   ONLINE       0     0     0
            c9t1d0   ONLINE       0     0     0
            c10t0d0  ONLINE       0     0     0
            c10t1d0  ONLINE       0     0     0
            c10t2d0  ONLINE       0     0     0
            c10t3d0  ONLINE       0     0     0
-- 
This message posted from opensolaris.org

Thanks for the tips guys, I''ll go with 2x 6drive raidz2 vdevs then.

Regards
Andreas Gunnarsson
-- 
This message posted from opensolaris.org