Hello, I''m about the build a opensolaris NAS system, currently we have two drives and are planning on adding two more at a later date (2TB enterprise level HDD are a bit expensive!). Whats the best configuration for setting up these drives bearing in mind I want to expand in the future? I was thinking of mirroring the drives and then converting to raidz some how? It will only be a max of 4 drives, the second two of which will be bought later. Any ideas? -- This message posted from opensolaris.org
On 7/14/10 Jul 14, 2:58 PM, "Daniel Taylor" <daniel at kaweb.co.uk> wrote:> I was thinking of mirroring the drives and then converting to raidz some how?Not possible. You can start with a mirror and then add another mirror; the filesystem will spread data across both drives in a way analogous* to RAID 10. *You can''t really compare ZFS to conventional RAID implementations, but if you look at it from 50,000 feet and squint you get the similarities. -- Dave Pooser, ACSA Manager of Information Services Alford Media http://www.alfordmedia.com
Hi Daniel,
No conversion from a mirrored to RAIDZ configuration is available yet.
Mirrored pools are more flexible and generally provide good performance.
You can easily create a mirrored pool of two disks and then add two
more disks later. You can also replace each disk with larger disks
if needed. See the example below.
If you consider that disks are relatively inexpensive, then a mirrored
configuration is a very good option.
Thanks,
Cindy
# zpool create mpool mirror c2t6d0 c2t7d0
# zpool list mpool
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
mpool 136G 78.5K 136G 0% 1.00x ONLINE -
# zpool status mpool
pool: mpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
mpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c2t6d0 ONLINE 0 0 0
c2t7d0 ONLINE 0 0 0
errors: No known data errors
# zpool add mpool mirror c2t8d0 c2t10d0
# zpool list mpool
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
mpool 272G 138K 272G 0% 1.00x ONLINE -
# zpool status mpool
pool: mpool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
mpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c2t6d0 ONLINE 0 0 0
c2t7d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c2t8d0 ONLINE 0 0 0
c2t10d0 ONLINE 0 0 0
errors: No known data errors
On 07/14/10 13:58, Daniel Taylor wrote:> Hello,
>
> I''m about the build a opensolaris NAS system, currently we have
two drives and are planning on adding two more at a later date (2TB enterprise
level HDD are a bit expensive!).
>
> Whats the best configuration for setting up these drives bearing in mind I
want to expand in the future?
>
> I was thinking of mirroring the drives and then converting to raidz some
how?
>
> It will only be a max of 4 drives, the second two of which will be bought
later.
>
> Any ideas?
Cindy, Thanks for getting back to me so quickly, I had though about that idea but (forgive me if I''m wrong) won''t that leave me with 4TB of space free from my (eventual) 8TB''s worth of drives? So 2 of the 2TB drives will be used just for mirroring. Where as with RaidZ I''d only lose one drive and have 3 usable (so 6TB, which is what I was going for) I can only fit 4 drives into the server chassis and I was hoping to get 6TB out of it. Thanks, - Daniel On 14 Jul 2010, at 21:28, Cindy Swearingen wrote:> Hi Daniel, > > No conversion from a mirrored to RAIDZ configuration is available yet. > > Mirrored pools are more flexible and generally provide good > performance. > > You can easily create a mirrored pool of two disks and then add two > more disks later. You can also replace each disk with larger disks > if needed. See the example below. > > If you consider that disks are relatively inexpensive, then a mirrored > configuration is a very good option. > > Thanks, > > Cindy > > # zpool create mpool mirror c2t6d0 c2t7d0 > # zpool list mpool > NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT > mpool 136G 78.5K 136G 0% 1.00x ONLINE - > # zpool status mpool > pool: mpool > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > mpool ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > c2t6d0 ONLINE 0 0 0 > c2t7d0 ONLINE 0 0 0 > > errors: No known data errors > # zpool add mpool mirror c2t8d0 c2t10d0 > # zpool list mpool > NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT > mpool 272G 138K 272G 0% 1.00x ONLINE - > # zpool status mpool > pool: mpool > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > mpool ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > c2t6d0 ONLINE 0 0 0 > c2t7d0 ONLINE 0 0 0 > mirror-1 ONLINE 0 0 0 > c2t8d0 ONLINE 0 0 0 > c2t10d0 ONLINE 0 0 0 > > errors: No known data errors > > > On 07/14/10 13:58, Daniel Taylor wrote: >> Hello, >> I''m about the build a opensolaris NAS system, currently we have two >> drives and are planning on adding two more at a later date (2TB >> enterprise level HDD are a bit expensive!). >> Whats the best configuration for setting up these drives bearing in >> mind I want to expand in the future? >> I was thinking of mirroring the drives and then converting to raidz >> some how? >> It will only be a max of 4 drives, the second two of which will be >> bought later. >> Any ideas?
Yes, that is true. If you have 4 2 TB drives, you would only get 4 TBs of usable disk space in a mirrored config. The problem I see with your potential RAIDZ config, if I understand it correctly, is that you can''t add more disks to an existing RAIDZ config. You would need to create a 3 disk RAIDZ1 config with 1 spare or a 4 disk RAIDZ1 config. You can''t attach more disks to an existing RAIDZ config. I understand that using all potential disk capacity is a priority, but I''ve been listening to this list for a long time and have come to appreciate the flexibility and reliability of mirrored configs. cs On 07/14/10 14:38, Daniel Taylor wrote:> Cindy, > > Thanks for getting back to me so quickly, > > I had though about that idea but (forgive me if I''m wrong) won''t that > leave me with 4TB of space free from my (eventual) 8TB''s worth of drives? > > So 2 of the 2TB drives will be used just for mirroring. Where as with > RaidZ I''d only lose one drive and have 3 usable (so 6TB, which is what I > was going for) > > I can only fit 4 drives into the server chassis and I was hoping to get > 6TB out of it. > > Thanks, > > - Daniel > > > On 14 Jul 2010, at 21:28, Cindy Swearingen wrote: > >> Hi Daniel, >> >> No conversion from a mirrored to RAIDZ configuration is available yet. >> >> Mirrored pools are more flexible and generally provide good performance. >> >> You can easily create a mirrored pool of two disks and then add two >> more disks later. You can also replace each disk with larger disks >> if needed. See the example below. >> >> If you consider that disks are relatively inexpensive, then a mirrored >> configuration is a very good option. >> >> Thanks, >> >> Cindy >> >> # zpool create mpool mirror c2t6d0 c2t7d0 >> # zpool list mpool >> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >> mpool 136G 78.5K 136G 0% 1.00x ONLINE - >> # zpool status mpool >> pool: mpool >> state: ONLINE >> scan: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> mpool ONLINE 0 0 0 >> mirror-0 ONLINE 0 0 0 >> c2t6d0 ONLINE 0 0 0 >> c2t7d0 ONLINE 0 0 0 >> >> errors: No known data errors >> # zpool add mpool mirror c2t8d0 c2t10d0 >> # zpool list mpool >> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >> mpool 272G 138K 272G 0% 1.00x ONLINE - >> # zpool status mpool >> pool: mpool >> state: ONLINE >> scan: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> mpool ONLINE 0 0 0 >> mirror-0 ONLINE 0 0 0 >> c2t6d0 ONLINE 0 0 0 >> c2t7d0 ONLINE 0 0 0 >> mirror-1 ONLINE 0 0 0 >> c2t8d0 ONLINE 0 0 0 >> c2t10d0 ONLINE 0 0 0 >> >> errors: No known data errors >> >> >> On 07/14/10 13:58, Daniel Taylor wrote: >>> Hello, >>> I''m about the build a opensolaris NAS system, currently we have two >>> drives and are planning on adding two more at a later date (2TB >>> enterprise level HDD are a bit expensive!). >>> Whats the best configuration for setting up these drives bearing in >>> mind I want to expand in the future? >>> I was thinking of mirroring the drives and then converting to raidz >>> some how? >>> It will only be a max of 4 drives, the second two of which will be >>> bought later. >>> Any ideas? >
David Dyer-Bennet
2010-Jul-14 21:44 UTC
[zfs-discuss] preparing for future drive additions
On Wed, July 14, 2010 14:58, Daniel Taylor wrote:> I''m about the build a opensolaris NAS system, currently we have two drives > and are planning on adding two more at a later date (2TB enterprise level > HDD are a bit expensive!).Do you really need them? Now? Maybe 1TB drives are good now, and then add a pair of 2TB in a year?> Whats the best configuration for setting up these drives bearing in mind I > want to expand in the future?Mirror now (pool consisting of one two-way mirror vdev). Add second mirror vdev to the pool when you need to expand.> I was thinking of mirroring the drives and then converting to raidz some > how?No way to convert to raidz. (That is, no magic simple way; you can of course put in new drives for the raidz and copy the data across.)> It will only be a max of 4 drives, the second two of which will be bought > later.5 drives would be a lot better. You could keep a hot spare -- and you could expand mirror vdevs safely (never dropping below your normal redundancy level), too. You can add new vdevs to a pool. This is very useful for a growing system (until you run out of drive slots). You can expand an existing vdev by replacing all the drives (one at a time). It''s a lot cleaner and safer with mirror vdevs than with raidz[ 23] vdevs. In a raid vdev, you can replace drives individually and wait for them to resilver. When each drive is done, replace the next. When you have replaced all of the drives, the vdev will then make the new space available. HOWEVER, doing this takes away a level of redundancy -- you take away a live drive. For a RAIDZ, that means no redundancy during the resilver (which takes a while on a 2TB drive, if you haven''t noticed). And the resilver is stressing the drives, so if there''s any incipient failure, it''s more likely to show up during the resilver. Scary! (RAIDZ2 is better in that you still have one layer of redundancy when you take one drive out; but in a 4-drive chassis forget it!). In a mirror vdev, you can be much cleverer, IF you can connect the new drive while the old drives are all still present. Attach the new bigger drive as a THIRD drive to the mirror vdev, and wait for the resilver. You now have a three-way mirror, and you never dropped below a two-way mirror at any time during the process. Detach one small drive and attach a new big drive, and wait again. And detach the last small drive, and you have now expanded your mirror vdev without ever dropping below your normal redundancy. (There are variants on this; the key point is that a mirror vdev can be an n-way mirror for any value of n your hardware can support.) If your backups are good and your uptime requirements aren''t really strict, of course the risks can be tolerated better. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
Cindy, Hmm, I''ve using RAIDZ-2 on a nexentastor elsewhere (although that has 15 disks) and having looked quite a bit I agree that mirroring is very flexible in comparison. I''ll have to think about it and see if I can afford to lose the 2TB''s. The other solution I can think off is to export the pool, redo everything with RAIDZ and then import the data? I presume that would work? But I would lose settings like samba shares? Thanks again! - Daniel On 14 Jul 2010, at 21:59, Cindy Swearingen wrote:> Yes, that is true. If you have 4 2 TB drives, you would only get 4 TBs > of usable disk space in a mirrored config. > > The problem I see with your potential RAIDZ config, if I understand > it correctly, is that you can''t add more disks to an existing RAIDZ > config. You would need to create a 3 disk RAIDZ1 config with 1 spare > or a 4 disk RAIDZ1 config. You can''t attach more disks to an existing > RAIDZ config. > > I understand that using all potential disk capacity is a priority, but > I''ve been listening to this list for a long time and have come to > appreciate the flexibility and reliability of mirrored configs. > > cs > > On 07/14/10 14:38, Daniel Taylor wrote: >> Cindy, >> Thanks for getting back to me so quickly, >> I had though about that idea but (forgive me if I''m wrong) won''t >> that leave me with 4TB of space free from my (eventual) 8TB''s worth >> of drives? >> So 2 of the 2TB drives will be used just for mirroring. Where as >> with RaidZ I''d only lose one drive and have 3 usable (so 6TB, which >> is what I was going for) >> I can only fit 4 drives into the server chassis and I was hoping to >> get 6TB out of it. >> Thanks, >> - Daniel >> On 14 Jul 2010, at 21:28, Cindy Swearingen wrote: >>> Hi Daniel, >>> >>> No conversion from a mirrored to RAIDZ configuration is available >>> yet. >>> >>> Mirrored pools are more flexible and generally provide good >>> performance. >>> >>> You can easily create a mirrored pool of two disks and then add two >>> more disks later. You can also replace each disk with larger disks >>> if needed. See the example below. >>> >>> If you consider that disks are relatively inexpensive, then a >>> mirrored >>> configuration is a very good option. >>> >>> Thanks, >>> >>> Cindy >>> >>> # zpool create mpool mirror c2t6d0 c2t7d0 >>> # zpool list mpool >>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>> mpool 136G 78.5K 136G 0% 1.00x ONLINE - >>> # zpool status mpool >>> pool: mpool >>> state: ONLINE >>> scan: none requested >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> mpool ONLINE 0 0 0 >>> mirror-0 ONLINE 0 0 0 >>> c2t6d0 ONLINE 0 0 0 >>> c2t7d0 ONLINE 0 0 0 >>> >>> errors: No known data errors >>> # zpool add mpool mirror c2t8d0 c2t10d0 >>> # zpool list mpool >>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>> mpool 272G 138K 272G 0% 1.00x ONLINE - >>> # zpool status mpool >>> pool: mpool >>> state: ONLINE >>> scan: none requested >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> mpool ONLINE 0 0 0 >>> mirror-0 ONLINE 0 0 0 >>> c2t6d0 ONLINE 0 0 0 >>> c2t7d0 ONLINE 0 0 0 >>> mirror-1 ONLINE 0 0 0 >>> c2t8d0 ONLINE 0 0 0 >>> c2t10d0 ONLINE 0 0 0 >>> >>> errors: No known data errors >>> >>> >>> On 07/14/10 13:58, Daniel Taylor wrote: >>>> Hello, >>>> I''m about the build a opensolaris NAS system, currently we have >>>> two drives and are planning on adding two more at a later date >>>> (2TB enterprise level HDD are a bit expensive!). >>>> Whats the best configuration for setting up these drives bearing >>>> in mind I want to expand in the future? >>>> I was thinking of mirroring the drives and then converting to >>>> raidz some how? >>>> It will only be a max of 4 drives, the second two of which will >>>> be bought later. >>>> Any ideas?
You can''t transition a mirrored pool to a RAIDZ pool with an pool export/import. If you find any info that says you can, please send a pointer. You would need to do these important middle steps: If you transition from a mirrored configuration to a RAIDZ configuration, you would need to backup the data, destroy the pool, recreate the pool as a RAIDZ configuration and restore the data. cs On 07/14/10 16:01, Daniel Taylor wrote:> Cindy, > > Hmm, I''ve using RAIDZ-2 on a nexentastor elsewhere (although that has 15 > disks) and having looked quite a bit I agree that mirroring is very > flexible in comparison. > > I''ll have to think about it and see if I can afford to lose the 2TB''s. > > The other solution I can think off is to export the pool, redo > everything with RAIDZ and then import the data? I presume that would > work? But I would lose settings like samba shares? > > Thanks again! > > - Daniel > On 14 Jul 2010, at 21:59, Cindy Swearingen wrote: > >> Yes, that is true. If you have 4 2 TB drives, you would only get 4 TBs >> of usable disk space in a mirrored config. >> >> The problem I see with your potential RAIDZ config, if I understand >> it correctly, is that you can''t add more disks to an existing RAIDZ >> config. You would need to create a 3 disk RAIDZ1 config with 1 spare >> or a 4 disk RAIDZ1 config. You can''t attach more disks to an existing >> RAIDZ config. >> >> I understand that using all potential disk capacity is a priority, but >> I''ve been listening to this list for a long time and have come to >> appreciate the flexibility and reliability of mirrored configs. >> >> cs >> >> On 07/14/10 14:38, Daniel Taylor wrote: >>> Cindy, >>> Thanks for getting back to me so quickly, >>> I had though about that idea but (forgive me if I''m wrong) won''t that >>> leave me with 4TB of space free from my (eventual) 8TB''s worth of >>> drives? >>> So 2 of the 2TB drives will be used just for mirroring. Where as with >>> RaidZ I''d only lose one drive and have 3 usable (so 6TB, which is >>> what I was going for) >>> I can only fit 4 drives into the server chassis and I was hoping to >>> get 6TB out of it. >>> Thanks, >>> - Daniel >>> On 14 Jul 2010, at 21:28, Cindy Swearingen wrote: >>>> Hi Daniel, >>>> >>>> No conversion from a mirrored to RAIDZ configuration is available yet. >>>> >>>> Mirrored pools are more flexible and generally provide good >>>> performance. >>>> >>>> You can easily create a mirrored pool of two disks and then add two >>>> more disks later. You can also replace each disk with larger disks >>>> if needed. See the example below. >>>> >>>> If you consider that disks are relatively inexpensive, then a mirrored >>>> configuration is a very good option. >>>> >>>> Thanks, >>>> >>>> Cindy >>>> >>>> # zpool create mpool mirror c2t6d0 c2t7d0 >>>> # zpool list mpool >>>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>>> mpool 136G 78.5K 136G 0% 1.00x ONLINE - >>>> # zpool status mpool >>>> pool: mpool >>>> state: ONLINE >>>> scan: none requested >>>> config: >>>> >>>> NAME STATE READ WRITE CKSUM >>>> mpool ONLINE 0 0 0 >>>> mirror-0 ONLINE 0 0 0 >>>> c2t6d0 ONLINE 0 0 0 >>>> c2t7d0 ONLINE 0 0 0 >>>> >>>> errors: No known data errors >>>> # zpool add mpool mirror c2t8d0 c2t10d0 >>>> # zpool list mpool >>>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>>> mpool 272G 138K 272G 0% 1.00x ONLINE - >>>> # zpool status mpool >>>> pool: mpool >>>> state: ONLINE >>>> scan: none requested >>>> config: >>>> >>>> NAME STATE READ WRITE CKSUM >>>> mpool ONLINE 0 0 0 >>>> mirror-0 ONLINE 0 0 0 >>>> c2t6d0 ONLINE 0 0 0 >>>> c2t7d0 ONLINE 0 0 0 >>>> mirror-1 ONLINE 0 0 0 >>>> c2t8d0 ONLINE 0 0 0 >>>> c2t10d0 ONLINE 0 0 0 >>>> >>>> errors: No known data errors >>>> >>>> >>>> On 07/14/10 13:58, Daniel Taylor wrote: >>>>> Hello, >>>>> I''m about the build a opensolaris NAS system, currently we have two >>>>> drives and are planning on adding two more at a later date (2TB >>>>> enterprise level HDD are a bit expensive!). >>>>> Whats the best configuration for setting up these drives bearing in >>>>> mind I want to expand in the future? >>>>> I was thinking of mirroring the drives and then converting to raidz >>>>> some how? >>>>> It will only be a max of 4 drives, the second two of which will be >>>>> bought later. >>>>> Any ideas? >
Sorry I think I used the wrong terms there, still learning, I think what I meant was send/receive rather than import/export. Would it be possible to use "zfs send" to backup the data somewhere (I''ll work that bit out later), then "receive" it into the new pool? And if I did that would I keep the snapshots? This system is going to be our backup storage NAS, so losing the snapshots is actually worse than losing the extra 2TB. Thanks, - Daniel On 14 Jul 2010, at 23:06, Cindy Swearingen wrote:> You can''t transition a mirrored pool to a RAIDZ pool with an > pool export/import. If you find any info that says you can, > please send a pointer. > > You would need to do these important middle steps: > > If you transition from a mirrored configuration to a RAIDZ > configuration, you would need to backup the data, destroy > the pool, recreate the pool as a RAIDZ configuration and > restore the data. > > cs > > On 07/14/10 16:01, Daniel Taylor wrote: >> Cindy, >> Hmm, I''ve using RAIDZ-2 on a nexentastor elsewhere (although that >> has 15 disks) and having looked quite a bit I agree that mirroring >> is very flexible in comparison. >> I''ll have to think about it and see if I can afford to lose the >> 2TB''s. >> The other solution I can think off is to export the pool, redo >> everything with RAIDZ and then import the data? I presume that >> would work? But I would lose settings like samba shares? >> Thanks again! >> - Daniel >> On 14 Jul 2010, at 21:59, Cindy Swearingen wrote: >>> Yes, that is true. If you have 4 2 TB drives, you would only get 4 >>> TBs >>> of usable disk space in a mirrored config. >>> >>> The problem I see with your potential RAIDZ config, if I understand >>> it correctly, is that you can''t add more disks to an existing >>> RAIDZ config. You would need to create a 3 disk RAIDZ1 config with >>> 1 spare >>> or a 4 disk RAIDZ1 config. You can''t attach more disks to an >>> existing >>> RAIDZ config. >>> >>> I understand that using all potential disk capacity is a priority, >>> but >>> I''ve been listening to this list for a long time and have come to >>> appreciate the flexibility and reliability of mirrored configs. >>> >>> cs >>> >>> On 07/14/10 14:38, Daniel Taylor wrote: >>>> Cindy, >>>> Thanks for getting back to me so quickly, >>>> I had though about that idea but (forgive me if I''m wrong) won''t >>>> that leave me with 4TB of space free from my (eventual) 8TB''s >>>> worth of drives? >>>> So 2 of the 2TB drives will be used just for mirroring. Where as >>>> with RaidZ I''d only lose one drive and have 3 usable (so 6TB, >>>> which is what I was going for) >>>> I can only fit 4 drives into the server chassis and I was hoping >>>> to get 6TB out of it. >>>> Thanks, >>>> - Daniel >>>> On 14 Jul 2010, at 21:28, Cindy Swearingen wrote: >>>>> Hi Daniel, >>>>> >>>>> No conversion from a mirrored to RAIDZ configuration is >>>>> available yet. >>>>> >>>>> Mirrored pools are more flexible and generally provide good >>>>> performance. >>>>> >>>>> You can easily create a mirrored pool of two disks and then add >>>>> two >>>>> more disks later. You can also replace each disk with larger disks >>>>> if needed. See the example below. >>>>> >>>>> If you consider that disks are relatively inexpensive, then a >>>>> mirrored >>>>> configuration is a very good option. >>>>> >>>>> Thanks, >>>>> >>>>> Cindy >>>>> >>>>> # zpool create mpool mirror c2t6d0 c2t7d0 >>>>> # zpool list mpool >>>>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>>>> mpool 136G 78.5K 136G 0% 1.00x ONLINE - >>>>> # zpool status mpool >>>>> pool: mpool >>>>> state: ONLINE >>>>> scan: none requested >>>>> config: >>>>> >>>>> NAME STATE READ WRITE CKSUM >>>>> mpool ONLINE 0 0 0 >>>>> mirror-0 ONLINE 0 0 0 >>>>> c2t6d0 ONLINE 0 0 0 >>>>> c2t7d0 ONLINE 0 0 0 >>>>> >>>>> errors: No known data errors >>>>> # zpool add mpool mirror c2t8d0 c2t10d0 >>>>> # zpool list mpool >>>>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>>>> mpool 272G 138K 272G 0% 1.00x ONLINE - >>>>> # zpool status mpool >>>>> pool: mpool >>>>> state: ONLINE >>>>> scan: none requested >>>>> config: >>>>> >>>>> NAME STATE READ WRITE CKSUM >>>>> mpool ONLINE 0 0 0 >>>>> mirror-0 ONLINE 0 0 0 >>>>> c2t6d0 ONLINE 0 0 0 >>>>> c2t7d0 ONLINE 0 0 0 >>>>> mirror-1 ONLINE 0 0 0 >>>>> c2t8d0 ONLINE 0 0 0 >>>>> c2t10d0 ONLINE 0 0 0 >>>>> >>>>> errors: No known data errors >>>>> >>>>> >>>>> On 07/14/10 13:58, Daniel Taylor wrote: >>>>>> Hello, >>>>>> I''m about the build a opensolaris NAS system, currently we have >>>>>> two drives and are planning on adding two more at a later date >>>>>> (2TB enterprise level HDD are a bit expensive!). >>>>>> Whats the best configuration for setting up these drives >>>>>> bearing in mind I want to expand in the future? >>>>>> I was thinking of mirroring the drives and then converting to >>>>>> raidz some how? >>>>>> It will only be a max of 4 drives, the second two of which will >>>>>> be bought later. >>>>>> Any ideas?
Yes, if you created snapshots of your file systems and stored them remotely, you could receive them into the new pool. I recommend that you test this process a few times before attempting the transition. Thanks, Cindy On 07/14/10 16:21, Daniel Taylor wrote:> Sorry I think I used the wrong terms there, still learning, I think what > I meant was send/receive rather than import/export. > > Would it be possible to use "zfs send" to backup the data somewhere > (I''ll work that bit out later), then "receive" it into the new pool? > > And if I did that would I keep the snapshots? > > This system is going to be our backup storage NAS, so losing the > snapshots is actually worse than losing the extra 2TB. > > Thanks, > > - Daniel > > On 14 Jul 2010, at 23:06, Cindy Swearingen wrote: > >> You can''t transition a mirrored pool to a RAIDZ pool with an >> pool export/import. If you find any info that says you can, >> please send a pointer. >> >> You would need to do these important middle steps: >> >> If you transition from a mirrored configuration to a RAIDZ >> configuration, you would need to backup the data, destroy >> the pool, recreate the pool as a RAIDZ configuration and >> restore the data. >> >> cs >> >> On 07/14/10 16:01, Daniel Taylor wrote: >>> Cindy, >>> Hmm, I''ve using RAIDZ-2 on a nexentastor elsewhere (although that has >>> 15 disks) and having looked quite a bit I agree that mirroring is >>> very flexible in comparison. >>> I''ll have to think about it and see if I can afford to lose the 2TB''s. >>> The other solution I can think off is to export the pool, redo >>> everything with RAIDZ and then import the data? I presume that would >>> work? But I would lose settings like samba shares? >>> Thanks again! >>> - Daniel >>> On 14 Jul 2010, at 21:59, Cindy Swearingen wrote: >>>> Yes, that is true. If you have 4 2 TB drives, you would only get 4 TBs >>>> of usable disk space in a mirrored config. >>>> >>>> The problem I see with your potential RAIDZ config, if I understand >>>> it correctly, is that you can''t add more disks to an existing RAIDZ >>>> config. You would need to create a 3 disk RAIDZ1 config with 1 spare >>>> or a 4 disk RAIDZ1 config. You can''t attach more disks to an existing >>>> RAIDZ config. >>>> >>>> I understand that using all potential disk capacity is a priority, but >>>> I''ve been listening to this list for a long time and have come to >>>> appreciate the flexibility and reliability of mirrored configs. >>>> >>>> cs >>>> >>>> On 07/14/10 14:38, Daniel Taylor wrote: >>>>> Cindy, >>>>> Thanks for getting back to me so quickly, >>>>> I had though about that idea but (forgive me if I''m wrong) won''t >>>>> that leave me with 4TB of space free from my (eventual) 8TB''s worth >>>>> of drives? >>>>> So 2 of the 2TB drives will be used just for mirroring. Where as >>>>> with RaidZ I''d only lose one drive and have 3 usable (so 6TB, which >>>>> is what I was going for) >>>>> I can only fit 4 drives into the server chassis and I was hoping to >>>>> get 6TB out of it. >>>>> Thanks, >>>>> - Daniel >>>>> On 14 Jul 2010, at 21:28, Cindy Swearingen wrote: >>>>>> Hi Daniel, >>>>>> >>>>>> No conversion from a mirrored to RAIDZ configuration is available >>>>>> yet. >>>>>> >>>>>> Mirrored pools are more flexible and generally provide good >>>>>> performance. >>>>>> >>>>>> You can easily create a mirrored pool of two disks and then add two >>>>>> more disks later. You can also replace each disk with larger disks >>>>>> if needed. See the example below. >>>>>> >>>>>> If you consider that disks are relatively inexpensive, then a >>>>>> mirrored >>>>>> configuration is a very good option. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Cindy >>>>>> >>>>>> # zpool create mpool mirror c2t6d0 c2t7d0 >>>>>> # zpool list mpool >>>>>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>>>>> mpool 136G 78.5K 136G 0% 1.00x ONLINE - >>>>>> # zpool status mpool >>>>>> pool: mpool >>>>>> state: ONLINE >>>>>> scan: none requested >>>>>> config: >>>>>> >>>>>> NAME STATE READ WRITE CKSUM >>>>>> mpool ONLINE 0 0 0 >>>>>> mirror-0 ONLINE 0 0 0 >>>>>> c2t6d0 ONLINE 0 0 0 >>>>>> c2t7d0 ONLINE 0 0 0 >>>>>> >>>>>> errors: No known data errors >>>>>> # zpool add mpool mirror c2t8d0 c2t10d0 >>>>>> # zpool list mpool >>>>>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>>>>> mpool 272G 138K 272G 0% 1.00x ONLINE - >>>>>> # zpool status mpool >>>>>> pool: mpool >>>>>> state: ONLINE >>>>>> scan: none requested >>>>>> config: >>>>>> >>>>>> NAME STATE READ WRITE CKSUM >>>>>> mpool ONLINE 0 0 0 >>>>>> mirror-0 ONLINE 0 0 0 >>>>>> c2t6d0 ONLINE 0 0 0 >>>>>> c2t7d0 ONLINE 0 0 0 >>>>>> mirror-1 ONLINE 0 0 0 >>>>>> c2t8d0 ONLINE 0 0 0 >>>>>> c2t10d0 ONLINE 0 0 0 >>>>>> >>>>>> errors: No known data errors >>>>>> >>>>>> >>>>>> On 07/14/10 13:58, Daniel Taylor wrote: >>>>>>> Hello, >>>>>>> I''m about the build a opensolaris NAS system, currently we have >>>>>>> two drives and are planning on adding two more at a later date >>>>>>> (2TB enterprise level HDD are a bit expensive!). >>>>>>> Whats the best configuration for setting up these drives bearing >>>>>>> in mind I want to expand in the future? >>>>>>> I was thinking of mirroring the drives and then converting to >>>>>>> raidz some how? >>>>>>> It will only be a max of 4 drives, the second two of which will >>>>>>> be bought later. >>>>>>> Any ideas? >
Perfect. Thank you you''ve been a great help, I have lots to think about (and test) now! Thanks again, nice to know this list is so responsive! - Daniel On 14 Jul 2010, at 23:34, Cindy Swearingen wrote:> Yes, if you created snapshots of your file systems and stored them > remotely, you could receive them into the new pool. > > I recommend that you test this process a few times before attempting > the transition. > > Thanks, > > Cindy > > On 07/14/10 16:21, Daniel Taylor wrote: >> Sorry I think I used the wrong terms there, still learning, I think >> what I meant was send/receive rather than import/export. >> Would it be possible to use "zfs send" to backup the data somewhere >> (I''ll work that bit out later), then "receive" it into the new pool? >> And if I did that would I keep the snapshots? >> This system is going to be our backup storage NAS, so losing the >> snapshots is actually worse than losing the extra 2TB. >> Thanks, >> - Daniel >> On 14 Jul 2010, at 23:06, Cindy Swearingen wrote: >>> You can''t transition a mirrored pool to a RAIDZ pool with an >>> pool export/import. If you find any info that says you can, >>> please send a pointer. >>> >>> You would need to do these important middle steps: >>> >>> If you transition from a mirrored configuration to a RAIDZ >>> configuration, you would need to backup the data, destroy >>> the pool, recreate the pool as a RAIDZ configuration and >>> restore the data. >>> >>> cs >>> >>> On 07/14/10 16:01, Daniel Taylor wrote: >>>> Cindy, >>>> Hmm, I''ve using RAIDZ-2 on a nexentastor elsewhere (although that >>>> has 15 disks) and having looked quite a bit I agree that >>>> mirroring is very flexible in comparison. >>>> I''ll have to think about it and see if I can afford to lose the >>>> 2TB''s. >>>> The other solution I can think off is to export the pool, redo >>>> everything with RAIDZ and then import the data? I presume that >>>> would work? But I would lose settings like samba shares? >>>> Thanks again! >>>> - Daniel >>>> On 14 Jul 2010, at 21:59, Cindy Swearingen wrote: >>>>> Yes, that is true. If you have 4 2 TB drives, you would only get >>>>> 4 TBs >>>>> of usable disk space in a mirrored config. >>>>> >>>>> The problem I see with your potential RAIDZ config, if I >>>>> understand >>>>> it correctly, is that you can''t add more disks to an existing >>>>> RAIDZ config. You would need to create a 3 disk RAIDZ1 config >>>>> with 1 spare >>>>> or a 4 disk RAIDZ1 config. You can''t attach more disks to an >>>>> existing >>>>> RAIDZ config. >>>>> >>>>> I understand that using all potential disk capacity is a >>>>> priority, but >>>>> I''ve been listening to this list for a long time and have come to >>>>> appreciate the flexibility and reliability of mirrored configs. >>>>> >>>>> cs >>>>> >>>>> On 07/14/10 14:38, Daniel Taylor wrote: >>>>>> Cindy, >>>>>> Thanks for getting back to me so quickly, >>>>>> I had though about that idea but (forgive me if I''m wrong) >>>>>> won''t that leave me with 4TB of space free from my (eventual) >>>>>> 8TB''s worth of drives? >>>>>> So 2 of the 2TB drives will be used just for mirroring. Where >>>>>> as with RaidZ I''d only lose one drive and have 3 usable (so >>>>>> 6TB, which is what I was going for) >>>>>> I can only fit 4 drives into the server chassis and I was >>>>>> hoping to get 6TB out of it. >>>>>> Thanks, >>>>>> - Daniel >>>>>> On 14 Jul 2010, at 21:28, Cindy Swearingen wrote: >>>>>>> Hi Daniel, >>>>>>> >>>>>>> No conversion from a mirrored to RAIDZ configuration is >>>>>>> available yet. >>>>>>> >>>>>>> Mirrored pools are more flexible and generally provide good >>>>>>> performance. >>>>>>> >>>>>>> You can easily create a mirrored pool of two disks and then >>>>>>> add two >>>>>>> more disks later. You can also replace each disk with larger >>>>>>> disks >>>>>>> if needed. See the example below. >>>>>>> >>>>>>> If you consider that disks are relatively inexpensive, then a >>>>>>> mirrored >>>>>>> configuration is a very good option. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Cindy >>>>>>> >>>>>>> # zpool create mpool mirror c2t6d0 c2t7d0 >>>>>>> # zpool list mpool >>>>>>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>>>>>> mpool 136G 78.5K 136G 0% 1.00x ONLINE - >>>>>>> # zpool status mpool >>>>>>> pool: mpool >>>>>>> state: ONLINE >>>>>>> scan: none requested >>>>>>> config: >>>>>>> >>>>>>> NAME STATE READ WRITE CKSUM >>>>>>> mpool ONLINE 0 0 0 >>>>>>> mirror-0 ONLINE 0 0 0 >>>>>>> c2t6d0 ONLINE 0 0 0 >>>>>>> c2t7d0 ONLINE 0 0 0 >>>>>>> >>>>>>> errors: No known data errors >>>>>>> # zpool add mpool mirror c2t8d0 c2t10d0 >>>>>>> # zpool list mpool >>>>>>> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT >>>>>>> mpool 272G 138K 272G 0% 1.00x ONLINE - >>>>>>> # zpool status mpool >>>>>>> pool: mpool >>>>>>> state: ONLINE >>>>>>> scan: none requested >>>>>>> config: >>>>>>> >>>>>>> NAME STATE READ WRITE CKSUM >>>>>>> mpool ONLINE 0 0 0 >>>>>>> mirror-0 ONLINE 0 0 0 >>>>>>> c2t6d0 ONLINE 0 0 0 >>>>>>> c2t7d0 ONLINE 0 0 0 >>>>>>> mirror-1 ONLINE 0 0 0 >>>>>>> c2t8d0 ONLINE 0 0 0 >>>>>>> c2t10d0 ONLINE 0 0 0 >>>>>>> >>>>>>> errors: No known data errors >>>>>>> >>>>>>> >>>>>>> On 07/14/10 13:58, Daniel Taylor wrote: >>>>>>>> Hello, >>>>>>>> I''m about the build a opensolaris NAS system, currently we >>>>>>>> have two drives and are planning on adding two more at a >>>>>>>> later date (2TB enterprise level HDD are a bit expensive!). >>>>>>>> Whats the best configuration for setting up these drives >>>>>>>> bearing in mind I want to expand in the future? >>>>>>>> I was thinking of mirroring the drives and then converting to >>>>>>>> raidz some how? >>>>>>>> It will only be a max of 4 drives, the second two of which >>>>>>>> will be bought later. >>>>>>>> Any ideas?
Cindy Swearingen wrote:> Hi Daniel, > > No conversion from a mirrored to RAIDZ configuration is available yet.Well... you can do it, but it''s a bit byzantine, and leaves you without redundancy during the migration. 1) Add your new disks 2) Create a sparse file at least as large as your smallest disk (via mkfile) 3) Scrub your pool (it had better be clean before you do this!) 4) Break the mirror 5) Create the new pool as raidz ${newdisks} ${ex-mirror} ${sparse file} 6) zpool offline the sparse file 7) zfs send/recv your old pool into your new pool 8) Verify all your zfs properties, data, etc. 9) zpool destroy your old pool 10) zpool replace the sparse file with the last disk 11) zpool export & re-import with the old pool name if desired -- Carson
Cindy wrote:> Mirrored pools are more flexible and generally > provide good performance. > > You can easily create a mirrored pool of two disks > and then add two > more disks later. You can also replace each disk with > larger disks > if needed. See the example below.There is no dispute that multiple vdevs (mirrors or otherwise) allow changing the drives in a single vdev without requiring a change the whole pool. There also is no dispute that mirrors provide better read iops than any other vdev type. On the other hand, situation after situation exists where 2+ drives offline in a pool leaving the RAIDZ1 and single mirror vdevs in real trouble. As I write this, the first thread in this forum is about an invalid pool because one drive died and another is offline, leaving the pool corrupted. This stuff just happens in the real world with non-DMX-class gear. One major point I read over and over about zfs was that it allowed the same level of protection without needing to spend $35 per GB of storage from an enterprise vendor. The only way to make this happen is with significant redundancy. I choose n+3 redundancy and love it. It''s like having two prebuilt hot spares. To achieve n+3 redundnancy with mirrors would require quadrupling the costs and spindle count vs. unprotected storage. It would seem that any vdev with n+1 protection is not adequate protection using sub million dollar storage equipment. -- This message posted from opensolaris.org
On Jul 14, 2010, at 11:44 PM, David Dyer-Bennet wrote:> On Wed, July 14, 2010 14:58, Daniel Taylor wrote: > >> I''m about the build a opensolaris NAS system, currently we have two drives >> and are planning on adding two more at a later date (2TB enterprise level >> HDD are a bit expensive!). > > Do you really need them? Now? Maybe 1TB drives are good now, and then > add a pair of 2TB in a year?I was recently at a large computer retailer and 1TB drives were not available for purchase. 2TB 3.5" for $110 and 500GB 2.5" drives were available. As David notes, if you plan to expand, plan to expand by replacing drives or adding pairs. This will be very cost efficient when your data space needs are modest. -- richard -- Richard Elling richard at nexenta.com +1-760-896-4422