Hi,
I''ve search the forums and not found any answer to the following.
I have 2 JBOD arrays each with 4 disks.
I want to create create a raidz on one array and have it mirrored to the other
array.
I''ve tried various command variations, but none seem to create what I
want.
e.g.
zpool create -f -m /export/data mzdata raidz c1t8d0 c1t9d0 c1t10d0 c1t11d0
mirror c2t8d0 c2t9d0 c1t10d0
Which created.... well, I''m not really sure.
But what I want is something like:
mzdataONLINE 0 0 0
mirror ONLINE 0 0 0
raidz ONLINE 0 0 0
c1t8d0 ONLINE 0 0 0
c1t9d0 ONLINE 0 0 0
c1t10d0 ONLINE 0 0 0
c1t11d0 ONLINE 0 0 0
raidz ONLINE 0 0 0
c1t15d0 ONLINE 0 0 0
c1t14d0 ONLINE 0 0 0
c1t13d0 ONLINE 0 0 0
c1t13d0 ONLINE 0 0 0
thanks for any help or advice.
_A_
This message posted from opensolaris.org
On October 19, 2006 9:02:47 PM -0700 Anthony Miller <anthonymiller1 at gmail.com> wrote:> Hi, > > I''ve search the forums and not found any answer to the following. > > I have 2 JBOD arrays each with 4 disks. > > I want to create create a raidz on one array and have it mirrored to the > other array.huh, why would you want raidz''s instead of just stripes. since you''re mirroring it anyway. zpool create mirror c0t0 c1t0 mirror c0t1 c1t1 ... which is like a raid10 (cf raid01) -frank
Anthony Miller wrote:> Hi, > > I''ve search the forums and not found any answer to the following. > > I have 2 JBOD arrays each with 4 disks. > > I want to create create a raidz on one array and have it mirrored to the other array.Do you think this will get you more availability compared to a simple mirror? I''m curious as to why you would want to do this.
On Oct 20, 2006, at 0:48, Torrey McMahon wrote:> Anthony Miller wrote: >> I want to create create a raidz on one array and have it mirrored to >> the other array. > > Do you think this will get you more availability compared to a simple > mirror? I''m curious as to why you would want to do this.This configuration will survive the failure of one drive in either RAIDZ *plus* the failure of any number of drives (or the whole mirror) in the other. That may or may not be valuable enough to choose, but it will survive more failures than just a mirror. --Ed
Anthony Miller wrote:> Hi, > > I''ve search the forums and not found any answer to the following. > > I have 2 JBOD arrays each with 4 disks. > > I want to create create a raidz on one array and have it mirrored to the other array.Today, the top level raid sets are assembled using dynamic striping. There is no option to assemble the sets with mirroring. Perhaps the ZFS team can enlighten us on their intentions in this area? -- richard
On October 20, 2006 8:43:03 AM -0700 Ed Gould <Ed.Gould at Sun.COM> wrote:> On Oct 20, 2006, at 0:48, Torrey McMahon wrote: >> Anthony Miller wrote: >>> I want to create create a raidz on one array and have it mirrored to >>> the other array. >> >> Do you think this will get you more availability compared to a simple >> mirror? I''m curious as to why you would want to do this. > > This configuration will survive the failure of one drive in either RAIDZ > *plus* the failure of any number of drives (or the whole mirror) in the > other. That may or may not be valuable enough to choose, but it will > survive more failures than just a mirror.Yes, it will survive more failures (one more, I think), but will that deliver more availability? Especially at the (high) cost of performance of a raidz vs a stripe. I''m also very curious to hear the reasoning. -frank
Frank Cusack wrote:> On October 20, 2006 8:43:03 AM -0700 Ed Gould <Ed.Gould at Sun.COM> wrote: >> On Oct 20, 2006, at 0:48, Torrey McMahon wrote: >>> Anthony Miller wrote: >>>> I want to create create a raidz on one array and have it mirrored to >>>> the other array. >>> >>> Do you think this will get you more availability compared to a simple >>> mirror? I''m curious as to why you would want to do this. >> >> This configuration will survive the failure of one drive in either RAIDZ >> *plus* the failure of any number of drives (or the whole mirror) in the >> other. That may or may not be valuable enough to choose, but it will >> survive more failures than just a mirror. > > Yes, it will survive more failures (one more, I think), but will that > deliver more availability? Especially at the (high) cost of performance > of a raidz vs a stripe.Yes, RAID-Z+mirror (or RAID-5+1) will provide better availability than RAID-Z+dynamic_stripe (or RAID-5+0). It will also provide better MTTDL. Random, small iop read performance will be similar to a pair of mirrored disks. -- richard
Don''t know if you are running current OpenSolaris or can wait for Solaris 10 11/06 (should be released in November). Either of those will contain raidz2 (which is like raid6 where you lose 2 disks). For max space with some redundancy, I would make one raidz2 vdev of all 8 disks. You will get the space of 6 disks, and can you can lose *any* 2 disks. Don''t know if that helps... This message posted from opensolaris.org
Richard Elling - PAE wrote:> Anthony Miller wrote: >> Hi, >> >> I''ve search the forums and not found any answer to the following. >> >> I have 2 JBOD arrays each with 4 disks. >> >> I want to create create a raidz on one array and have it mirrored to >> the other array. > > Today, the top level raid sets are assembled using dynamic striping. There > is no option to assemble the sets with mirroring. Perhaps the ZFS team can > enlighten us on their intentions in this area?Our thinking is that if you want more redundancy than RAID-Z, you should use RAID-Z with double parity, which provides more reliability and more usable storage than a mirror of RAID-Zs would. (Also, expressing "mirror of RAID-Zs" from the CLI would be a bit messy; you''d have to introduce parentheses in vdev descriptions or something.) --matt
Matthew Ahrens a ?crit :> Richard Elling - PAE wrote: > >> Anthony Miller wrote: >> >>> Hi, >>> >>> I''ve search the forums and not found any answer to the following. >>> >>> I have 2 JBOD arrays each with 4 disks. >>> >>> I want to create create a raidz on one array and have it mirrored to >>> the other array. >> >> >> Today, the top level raid sets are assembled using dynamic striping. >> There >> is no option to assemble the sets with mirroring. Perhaps the ZFS >> team can >> enlighten us on their intentions in this area? > > > Our thinking is that if you want more redundancy than RAID-Z, you > should use RAID-Z with double parity, which provides more reliability > and more usable storage than a mirror of RAID-Zs would. > > (Also, expressing "mirror of RAID-Zs" from the CLI would be a bit > messy; you''d have to introduce parentheses in vdev descriptions or > something.)It is not always a matter of more redundancy. In my customer''s case, they have storage in 2 different rooms of their datacenter and want to mirror from one storage unit in one room to the other. So having in this case a combination of RAID-Z + Mirror makes sense in my mind .... or ? Michel.
Michel Kintz writes: > Matthew Ahrens a ?crit : > > > Richard Elling - PAE wrote: > > > >> Anthony Miller wrote: > >> > >>> Hi, > >>> > >>> I''ve search the forums and not found any answer to the following. > >>> > >>> I have 2 JBOD arrays each with 4 disks. > >>> > >>> I want to create create a raidz on one array and have it mirrored to > >>> the other array. > >> > >> > >> Today, the top level raid sets are assembled using dynamic striping. > >> There > >> is no option to assemble the sets with mirroring. Perhaps the ZFS > >> team can > >> enlighten us on their intentions in this area? > > > > > > Our thinking is that if you want more redundancy than RAID-Z, you > > should use RAID-Z with double parity, which provides more reliability > > and more usable storage than a mirror of RAID-Zs would. > > > > (Also, expressing "mirror of RAID-Zs" from the CLI would be a bit > > messy; you''d have to introduce parentheses in vdev descriptions or > > something.) > > It is not always a matter of more redundancy. > In my customer''s case, they have storage in 2 different rooms of their > datacenter and want to mirror from one storage unit in one room to the > other. > So having in this case a combination of RAID-Z + Mirror makes sense in > my mind .... or ? > > Michel. > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss you may let the storage export RAID-5 luns and let ZFS mirror those. Would that work ? -r
On Oct 24, 2006, at 4:56 AM, Michel Kintz wrote:> It is not always a matter of more redundancy. > In my customer''s case, they have storage in 2 different rooms of > their datacenter and want to mirror from one storage unit in one > room to the other. > So having in this case a combination of RAID-Z + Mirror makes sense > in my mind .... or ?It /does/ make sense. Having a geographically diverse storage scenario like this is good, but changes the rules a bit, and in a way that you can''t fully take advantage of by using only soft RAID such as ZFS or SVM. The missing link as you point out is the missing ability to mirror (within the ZFS) a RAIDZ vdev. To get around this, I just use hardware RAID5 on my separate arrays and use either ZFS or SVM mirroring between the two on the hosts. I have thought about this over the past several months, and believe that it''s probably better this way rather doing it all in ZFS or SVM. /dale
On Oct 24, 2006, at 04:19, Roch wrote:> > Michel Kintz writes: >> Matthew Ahrens a ?crit : >> >>> Richard Elling - PAE wrote: >>> >>>> Anthony Miller wrote: >>>> >>>>> Hi, >>>>> >>>>> I''ve search the forums and not found any answer to the following. >>>>> >>>>> I have 2 JBOD arrays each with 4 disks. >>>>> >>>>> I want to create create a raidz on one array and have it >>>>> mirrored to >>>>> the other array. >>>> >>>> >>>> Today, the top level raid sets are assembled using dynamic >>>> striping. >>>> There >>>> is no option to assemble the sets with mirroring. Perhaps the ZFS >>>> team can >>>> enlighten us on their intentions in this area? >>> >>> >>> Our thinking is that if you want more redundancy than RAID-Z, you >>> should use RAID-Z with double parity, which provides more >>> reliability >>> and more usable storage than a mirror of RAID-Zs would. >>> >>> (Also, expressing "mirror of RAID-Zs" from the CLI would be a bit >>> messy; you''d have to introduce parentheses in vdev descriptions or >>> something.) >> >> It is not always a matter of more redundancy. >> In my customer''s case, they have storage in 2 different rooms of >> their >> datacenter and want to mirror from one storage unit in one room to >> the >> other. >> So having in this case a combination of RAID-Z + Mirror makes >> sense in >> my mind .... or ? >> >> Michel. >> >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > you may let the storage export RAID-5 luns and let ZFS > mirror those. Would that work ? > > -rthey''re JBOD arrays, so unless you''re proposing the use of another volume manager i don''t think that would work. as for the maximum redundancy in configurations, i think that Frank hit it with the mirroring of each drive component across the arrays and doing a simple stripe I just think it would be good to add the flexibility in zpool to: 1) raidz a set of mirrors 2) mirror a couple of raidz in certain environments you care more about multiple drive or array failures than anything else. Today you can do this with zvols, but I''m a little worried about how this would perform given the nested layering you have to introduce .. eg: # zpool create a1pool raidz c0t0d0 c0t1d0 c0t2d0 .. # zpool create a2pool raidz c1t0d0 c1t1d0 c1t2d0 .. # zfs create -V <size> a1pool/vol # zfs create -V <size> a2pool/vol # zpool create mzdata mirror /dev/zvol/dsk/a1pool/vol /dev/zvol/dsk/ a2pool/vol .je
>Our thinking is that if you want more redundancy than RAID-Z, you should >use RAID-Z with double parity, which provides more reliability and more >usable storage than a mirror of RAID-Zs would.This is only true if the drives have either independent or identical failure modes, I think. Consider two boxes, each containing ten drives. Creating RAID-Z within each box protects against single-drive failures. Mirroring the boxes together protects against single-box failures.>(Also, expressing "mirror of RAID-Zs" from the CLI would be a bit messy; >you''d have to introduce parentheses in vdev descriptions or something.)That doesn''t sound so bad, actually. (An alternative would be to take the SVM approach and allow vdevs to be built up in multiple commands.) This message posted from opensolaris.org
On October 24, 2006 9:19:07 AM -0700 "Anton B. Rang" <Anton.Rang at Sun.COM> wrote:>> Our thinking is that if you want more redundancy than RAID-Z, you should >> use RAID-Z with double parity, which provides more reliability and more >> usable storage than a mirror of RAID-Zs would. > > This is only true if the drives have either independent or identical > failure modes, I think. Consider two boxes, each containing ten drives. > Creating RAID-Z within each box protects against single-drive failures. > Mirroring the boxes together protects against single-box failures.But mirroring also protects against single-drive failures. -frank
On Oct 24, 2006, at 12:33 PM, Frank Cusack wrote:> On October 24, 2006 9:19:07 AM -0700 "Anton B. Rang" > <Anton.Rang at Sun.COM> wrote: >>> Our thinking is that if you want more redundancy than RAID-Z, you >>> should >>> use RAID-Z with double parity, which provides more reliability >>> and more >>> usable storage than a mirror of RAID-Zs would. >> >> This is only true if the drives have either independent or identical >> failure modes, I think. Consider two boxes, each containing ten >> drives. >> Creating RAID-Z within each box protects against single-drive >> failures. >> Mirroring the boxes together protects against single-box failures. > > But mirroring also protects against single-drive failures.Right, but mirrored raidz would in this case protect the admin from: 1) one entire jbod chassis/comm failure, and 2) individual drive failure in the remaining chassis during an occurrence of (1) Since the person is dealing with JBODS and not hardware RAID arrays, my suggestion is to combine ZFS and SVM. 1) Use ZFS and make a raidz-based ZVOL of disks on each of the two JBODs 2) Use SVM to mirror the two ZVOLs. Newfs that with UFS. Not at all optimal, but it''ll work. It would be nice if you could manage a mirror of existing vdevs within ZFS and this mirroring would be a special case where it would be dumb and just present the volume and pass through most of the stuff to the raidz (or whatever) vdev below. It would be silly to double-cksum and compress everything, not to mention the possibility of differing record sizes. /dale
On October 24, 2006 2:26:49 PM -0400 Dale Ghent <daleg at elemental.org> wrote:> Since the person is dealing with JBODS and not hardware RAID arrays, my > suggestion is to combine ZFS and SVM. > > 1) Use ZFS and make a raidz-based ZVOL of disks on each of the two JBODs > 2) Use SVM to mirror the two ZVOLs. Newfs that with UFS.Why wouldn''t you use a zfs mirror on top of the two zvol''s? -frank
On Oct 24, 2006, at 12:26, Dale Ghent wrote:> On Oct 24, 2006, at 12:33 PM, Frank Cusack wrote: > >> On October 24, 2006 9:19:07 AM -0700 "Anton B. Rang" >> <Anton.Rang at Sun.COM> wrote: >>>> Our thinking is that if you want more redundancy than RAID-Z, >>>> you should >>>> use RAID-Z with double parity, which provides more reliability >>>> and more >>>> usable storage than a mirror of RAID-Zs would. >>> >>> This is only true if the drives have either independent or identical >>> failure modes, I think. Consider two boxes, each containing ten >>> drives. >>> Creating RAID-Z within each box protects against single-drive >>> failures. >>> Mirroring the boxes together protects against single-box failures. >> >> But mirroring also protects against single-drive failures. > > Right, but mirrored raidz would in this case protect the admin from: > > 1) one entire jbod chassis/comm failure, and > 2) individual drive failure in the remaining chassis during an > occurrence of (1) > > Since the person is dealing with JBODS and not hardware RAID > arrays, my suggestion is to combine ZFS and SVM. > > 1) Use ZFS and make a raidz-based ZVOL of disks on each of the two > JBODs > 2) Use SVM to mirror the two ZVOLs. Newfs that with UFS. > > Not at all optimal, but it''ll work. It would be nice if you could > manage a mirror of existing vdevs within ZFS and this mirroring > would be a special case where it would be dumb and just present the > volume and pass through most of the stuff to the raidz (or > whatever) vdev below. It would be silly to double-cksum and > compress everything, not to mention the possibility of differing > record sizes.RFE submitted as 6485689 .je
Pedantic question, what would this gain us other than better data retention? Space and (especially?) performance would be worse with RAID-Z+1 than 2-way mirrors. -- richard Frank Cusack wrote:> On October 24, 2006 9:19:07 AM -0700 "Anton B. Rang" > <Anton.Rang at Sun.COM> wrote: >>> Our thinking is that if you want more redundancy than RAID-Z, you should >>> use RAID-Z with double parity, which provides more reliability and more >>> usable storage than a mirror of RAID-Zs would. >> >> This is only true if the drives have either independent or identical >> failure modes, I think. Consider two boxes, each containing ten drives. >> Creating RAID-Z within each box protects against single-drive failures. >> Mirroring the boxes together protects against single-box failures. > > But mirroring also protects against single-drive failures. > > -frank > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Oct 24, 2006, at 2:46 PM, Richard Elling - PAE wrote:> Pedantic question, what would this gain us other than better data > retention? > Space and (especially?) performance would be worse with RAID-Z+1 > than 2-way mirrors.You answered your own question, it would gain the user better data retention :) The space tradeoff is an obvious side effect and unavoidable. For situations where this is not an overriding issue, it just isn''t an issue. I don''t believe performance would be adversely impacted to a practical degree, though. A "dumb" ZFS mirror strategy in this case would just copy reads and writes to and from the vdevs below it, OR pre-package the writes itself with compression and checksums and send that data below to the raidz''s to be stored (which would probably be more problematic to implement in the zfs code). With the latter, checksums and compression would be done only once (at the mirror level) and not done by each of the n number of underlying vdevs. So, a little ascii art to summarize: 1) The probably-easiest-to-implement approach: [app] | [zfs volume] | [vdev mirror] <-- passes thru read/write ops, regulates recordsize. It''s mainly "dumb" | [raidz vdev]-- |--[raidz vdev]... <-- each vdev generates cksums, compression per normal | | | | | | | | [phys devs] [phys devs] 2) The less-CPU-but-more-convoluted approach: [app] | [zfs volume] | [vdev mirror] <-- generates cksums, compression, regulates recordsize | [raidz vdev]-- |--[raidz vdev]... <-- each vdev just stores data as it is passed in from above | | | | | | | | [phys devs] [phys devs] Any of those two would be quite handy in the environment where you want to mirror data between, say, JBODs and retention is the primary goal. /dale
there''s 2 approaches: 1) RAID 1+Z where you mirror the individual drives across trays and then RAID-Z the whole thing 2) RAID Z+1 where you RAIDZ each tray and then mirror them I would argue that you can lose the most drives in configuration 1 and stay alive: With a simple mirrored stripe you lose if you lose 1 drives in each tray. With configuration 2 this takes it 2 drives in each tray. With configuration 1 you have to lose both sides of a 2 mirrored sets to fail. so it''s not a space or performance model .. simply an availability model with failing disk Jonathan On Oct 24, 2006, at 12:46, Richard Elling - PAE wrote:> Pedantic question, what would this gain us other than better data > retention? > Space and (especially?) performance would be worse with RAID-Z+1 > than 2-way mirrors. > -- richard > > Frank Cusack wrote: >> On October 24, 2006 9:19:07 AM -0700 "Anton B. Rang" >> <Anton.Rang at Sun.COM> wrote: >>>> Our thinking is that if you want more redundancy than RAID-Z, >>>> you should >>>> use RAID-Z with double parity, which provides more reliability >>>> and more >>>> usable storage than a mirror of RAID-Zs would. >>> >>> This is only true if the drives have either independent or identical >>> failure modes, I think. Consider two boxes, each containing ten >>> drives. >>> Creating RAID-Z within each box protects against single-drive >>> failures. >>> Mirroring the boxes together protects against single-box failures. >> But mirroring also protects against single-drive failures. >> -frank >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On October 24, 2006 3:15:10 PM -0400 Dale Ghent <daleg at elemental.org> wrote:> On Oct 24, 2006, at 2:46 PM, Richard Elling - PAE wrote: > >> Pedantic question, what would this gain us other than better data >> retention? >> Space and (especially?) performance would be worse with RAID-Z+1 >> than 2-way mirrors. > > You answered your own question, it would gain the user better data > retention :) > > The space tradeoff is an obvious side effect and unavoidable. For > situations where this is not an overriding issue, it just isn''t an > issue. I don''t believe performance would be adversely impacted to a > practical degree, though.Really? <http://blogs.sun.com/roch/entry/when_to_and_not_to> says a raid-z vdev has the read throughput of 1 drive for random reads. Compared to #drives for a stripe. That''s pretty significant. -frank
On Oct 24, 2006, at 3:23 PM, Frank Cusack wrote:> <http://blogs.sun.com/roch/entry/when_to_and_not_to> says a raid-z > vdev has the read throughput of 1 drive for random reads. Compared > to #drives for a stripe. That''s pretty significant.Okay, then if the person can stand to lose even more space, do zfs mirroring on each JBOD. Then we''d have a mirror of mirrors instead of a mirror of raidz''s. Remember, the OP wanted chassis-level redundancy as well as redundancy within the domain of each chassis. You can''t do that now with ZFS unless you combine ZFS with SVM. /dale
On October 24, 2006 3:31:41 PM -0400 Dale Ghent <daleg at elemental.org> wrote:> On Oct 24, 2006, at 3:23 PM, Frank Cusack wrote: > >> <http://blogs.sun.com/roch/entry/when_to_and_not_to> says a raid-z >> vdev has the read throughput of 1 drive for random reads. Compared >> to #drives for a stripe. That''s pretty significant. > > Okay, then if the person can stand to lose even more space, do zfs > mirroring on each JBOD. Then we''d have a mirror of mirrors instead of a > mirror of raidz''s. > > Remember, the OP wanted chassis-level redundancy as well as redundancy > within the domain of each chassis. You can''t do that now with ZFS unless > you combine ZFS with SVM.I don''t think we know what the OP wanted. :-) -frank
Frank Cusack wrote:> > I don''t think we know what the OP wanted. :-)I understand the paranoia around overlapping raid levels - And yes they are out to get you - but in the past some of the requirements were around performance in a failure mode. Do we have any data concerning the performance of a degraded RAIDZ volume? This might alleviate the fears of some....even though we are out to get you. Wait...did I see we? :) -- Torrey McMahon Sun Microsystems Inc.
On 10/25/06, Jonathan Edwards <Jonathan.Edwards at sun.com> wrote:> > On Oct 24, 2006, at 12:26, Dale Ghent wrote: > > > On Oct 24, 2006, at 12:33 PM, Frank Cusack wrote: > > > >> On October 24, 2006 9:19:07 AM -0700 "Anton B. Rang" > >> <Anton.Rang at Sun.COM> wrote: > >>>> Our thinking is that if you want more redundancy than RAID-Z, > >>>> you should > >>>> use RAID-Z with double parity, which provides more reliability > >>>> and more > >>>> usable storage than a mirror of RAID-Zs would. > >>> > >>> This is only true if the drives have either independent or identical > >>> failure modes, I think. Consider two boxes, each containing ten > >>> drives. > >>> Creating RAID-Z within each box protects against single-drive > >>> failures. > >>> Mirroring the boxes together protects against single-box failures. > >> > >> But mirroring also protects against single-drive failures. > > > > Right, but mirrored raidz would in this case protect the admin from: > > > > 1) one entire jbod chassis/comm failure, and > > 2) individual drive failure in the remaining chassis during an > > occurrence of (1) > > > > Since the person is dealing with JBODS and not hardware RAID > > arrays, my suggestion is to combine ZFS and SVM. > > > > 1) Use ZFS and make a raidz-based ZVOL of disks on each of the two > > JBODs > > 2) Use SVM to mirror the two ZVOLs. Newfs that with UFS. > > > > Not at all optimal, but it''ll work. It would be nice if you could > > manage a mirror of existing vdevs within ZFS and this mirroring > > would be a special case where it would be dumb and just present the > > volume and pass through most of the stuff to the raidz (or > > whatever) vdev below. It would be silly to double-cksum and > > compress everything, not to mention the possibility of differing > > record sizes. > > RFE submitted as 6485689I''m willing to work on this if no one else is. :) -- Regards, Jeremy
On Tue, Oct 24, 2006 at 03:31:41PM -0400, Dale Ghent wrote:> Okay, then if the person can stand to lose even more space, do zfs > mirroring on each JBOD. Then we''d have a mirror of mirrors instead of > a mirror of raidz''s. > > Remember, the OP wanted chassis-level redundancy as well as > redundancy within the domain of each chassis. You can''t do that now > with ZFS unless you combine ZFS with SVM.If you have the disk space to let you do a mirror of mirrors, you could create the pool from a series of 4 way mirrors, with each mirror containing 2 disks from each JBOD enclosure. This would give you the ability to sustain the simultaneous failure of an entire enclosure and at least one (though sometimes multiple) disk failure in the working enclosure. This would also be a pure ZFS solution. Ed Plese