Hi all,
Yet another question regarding raidz configuration..
Assuming a system with 24 disks available , having in mind reliability
as the crucial factor , secondary the usable space and finally
performance would be the last criteria, what would be the preferable
configuration ?
Should it be :
* A - 7 raidz2 groups with 3 disks each and 3 disks as hot-spares
* B - 3 raidz2 groups with 7 disks each and 3 disks as hot-spares
* C - 5 raidz2 groups with 4 disks each and 4 disks as hot-spares
* D - 4 raidz2 groups with 5 disks each and 4 disks as hot-spares
* E - other? ;)
Thanks for all your attention,
Bruno
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/506dace2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3656 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/506dace2/attachment.bin>
I think I would do 3xraidz3 with 8 disks and 0 hotspares. That way you have a better chance of resolving bit rot issues that might become apparent during a rebuild. -- This message posted from opensolaris.org
On Thu, Mar 25, 2010 at 6:28 AM, Bruno Sousa <bsousa at epinfante.com> wrote:> Assuming a system with 24 disks available , having in mind reliability as > the crucial factor , secondary the usable space and finally performance > would be the last criteria, what would be the preferable configuration ? > > Should it be : > > - A - 7 raidz2 groups with 3 disks each and 3 disks as hot-spares > - B - 3 raidz2 groups with 7 disks each and 3 disks as hot-spares > - C - 5 raidz2 groups with 4 disks each and 4 disks as hot-spares > - D - 4 raidz2 groups with 5 disks each and 4 disks as hot-spares > - E - other? ;) > > Using fewer than 4 disks in a raidz2 defeats the purpose of raidz2, as youwill always be in a degraded mode. Why do you want so many hot-spares? Are you really expecting that many drives to die simultaneously? 1-2 would be plenty for a 24 drive system, if even that many. We have a couple 24-drive storage servers running. They are currently using 3x raidz2 vdevs of 8 drives each, spread across two 12-port controllers (8 on one controller, 8 on the other, 8 spread across the two). If I was to re-do these servers today, I would go with 4x raidz2 vdevs of 6 drives each and either put 2 vdevs on each controller, or switch to using 4 separate 8-port controllers with 1 vdev per controller. We don''t use hot-spares, as we have a lot of monitoring to detect when a drive dies or the pool becomes degraded, and have a stack of spare drives standing by to use as replacements. -- Freddie Cash fjwcash at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/fdaa253e/attachment.html>
Hi, What do you mean by "Using fewer than 4 disks in a raidz2 defeats the purpose of raidz2, as you will always be in a degraded mode" ? Does it means that having 2 vdevs with 3 disks it won''t be redundant in the advent of a drive failure? Indeed it may be too many spares...the discussion here it''s between myself (the new guy that embraces new technologies) vs a EMC "lover" sysadmin... Currently the system is configured with 4 vdevs each with 5 drives, and i''m considering to use the 4 others ones to create something a raid10 (usefull for the backup software indexes...) . So far the system seems to behave quite nice...but than again we are just starting it. Thanks for the input, Bruno On 25-3-2010 16:46, Freddie Cash wrote:> On Thu, Mar 25, 2010 at 6:28 AM, Bruno Sousa <bsousa at epinfante.com > <mailto:bsousa at epinfante.com>> wrote: > > Assuming a system with 24 disks available , having in mind > reliability as the crucial factor , secondary the usable space and > finally performance would be the last criteria, what would be the > preferable configuration ? > > Should it be : > > * A - 7 raidz2 groups with 3 disks each and 3 disks as hot-spares > * B - 3 raidz2 groups with 7 disks each and 3 disks as hot-spares > * C - 5 raidz2 groups with 4 disks each and 4 disks as hot-spares > * D - 4 raidz2 groups with 5 disks each and 4 disks as hot-spares > * E - other? ;) > > Using fewer than 4 disks in a raidz2 defeats the purpose of raidz2, as > you will always be in a degraded mode. > > Why do you want so many hot-spares? Are you really expecting that > many drives to die simultaneously? 1-2 would be plenty for a 24 drive > system, if even that many. > > We have a couple 24-drive storage servers running. They are currently > using 3x raidz2 vdevs of 8 drives each, spread across two 12-port > controllers (8 on one controller, 8 on the other, 8 spread across the > two). > > If I was to re-do these servers today, I would go with 4x raidz2 vdevs > of 6 drives each and either put 2 vdevs on each controller, or switch > to using 4 separate 8-port controllers with 1 vdev per controller. > > We don''t use hot-spares, as we have a lot of monitoring to detect when > a drive dies or the pool becomes degraded, and have a stack of spare > drives standing by to use as replacements. > > -- > Freddie Cash > fjwcash at gmail.com <mailto:fjwcash at gmail.com> > > -- > This message has been scanned for viruses and > dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is > believed to be clean. > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/8fa758e9/attachment.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3656 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/8fa758e9/attachment.bin>
On 25-3-2010 15:28, Richard Jahnel wrote:> I think I would do 3xraidz3 with 8 disks and 0 hotspares. > > That way you have a better chance of resolving bit rot issues that might become apparent during a rebuild. >Indeed raidz3...i didn''t consider it. In short, a raidz3 could sustain 3 broken drives per vdev vs 2 broken drives in a raidz2 pool ? Perfomance and usable space , how does it compares? Bruno -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3656 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/281b6e33/attachment.bin>
Bruno Sousa wrote:> What do you mean by "Using fewer than 4 disks in a raidz2 defeats the > purpose of raidz2, as you will always be in a degraded mode" ? > Does it means that having 2 vdevs with 3 disks it won''t be redundant in > the advent of a drive failure?Technically a 3 disk raidz2 won''t be degraded, so I''m not sure what he meant. However a 3-way mirror will be much faster and equally reliable, so a 3 disk raidz2 doesn''t make much sense. -- Carson
Well the thing I like about raidz3 is that even with 1 drive out you have 3 copies of all the blocks. So if you encounter bit rot, not only can checksums be used to find the good data, you can still get a best 2 out of 3 vote on which data is correct. As to performance, all I can say is test test test. Pick your top 3 contenders, install bonnie++ and then test each configuration. Then make your decision based on your personal balance between performance, space and reliability. Due to space considerations I had to choose raidz2 over raidz3. I just couldn''t give up that last drives worth of space. oddly enough in my enviroment I got better performance out of raidz2 than I did out of 7 mirrors striped together. It may be because they are all 250gb ssds. I think the bottleneck in my case is either the Adaptec raid card or cpu. -- This message posted from opensolaris.org
On Thu, Mar 25, 2010 at 11:47 AM, Bruno Sousa <bsousa at epinfante.com> wrote:> What do you mean by "Using fewer than 4 disks in a raidz2 defeats the > purpose of raidz2, as you will always be in a degraded mode" ? Does it means > that having 2 vdevs with 3 disks it won''t be redundant in the advent of a > drive failure? >raidz1 is similar to raid5 in that it is single-parity, and requires a minimum of 3 drives (2 data + 1 parity) raidz2 is similar to raid6 in that it is double-parity, and requires a minimum of 4 drives (2 data + 2 parity) IOW, a raidz2 vdev made up of 3 drives will always be running in degraded mode (it''s missing a drive). -- Freddie Cash fjwcash at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/5a3b8bf9/attachment.html>
Hmm...it might be completely wrong , but the idea of raidz2 vdev with 3
disks came from the reading of
http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view .
This particular page has the following example :
*zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0*
# *zpool status -v tank*
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c3t0d0 ONLINE 0 0 0
So...what am i missing here? Just a bad example in the sun documentation
regarding zfs?
Bruno
On 25-3-2010 20:10, Freddie Cash wrote:> On Thu, Mar 25, 2010 at 11:47 AM, Bruno Sousa <bsousa at epinfante.com
> <mailto:bsousa at epinfante.com>> wrote:
>
> What do you mean by "Using fewer than 4 disks in a raidz2 defeats
> the purpose of raidz2, as you will always be in a degraded mode" ?
> Does it means that having 2 vdevs with 3 disks it won''t be
> redundant in the advent of a drive failure?
>
>
> raidz1 is similar to raid5 in that it is single-parity, and requires a
> minimum of 3 drives (2 data + 1 parity)
> raidz2 is similar to raid6 in that it is double-parity, and requires a
> minimum of 4 drives (2 data + 2 parity)
>
> IOW, a raidz2 vdev made up of 3 drives will always be running in
> degraded mode (it''s missing a drive).
>
> --
> Freddie Cash
> fjwcash at gmail.com <mailto:fjwcash at gmail.com>
>
> --
> This message has been scanned for viruses and
> dangerous content by *MailScanner* <http://www.mailscanner.info/>,
and is
> believed to be clean.
>
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/3203cfd4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3656 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/3203cfd4/attachment.bin>
On Mar 25, 2010, at 22:10, Freddie Cash <fjwcash at gmail.com> wrote:> On Thu, Mar 25, 2010 at 11:47 AM, Bruno Sousa <bsousa at epinfante.com> > wrote: > What do you mean by "Using fewer than 4 disks in a raidz2 defeats > the purpose of raidz2, as you will always be in a degraded mode" ? > Does it means that having 2 vdevs with 3 disks it won''t be redundant > in the advent of a drive failure? > > raidz1 is similar to raid5 in that it is single-parity, and requires > a minimum of 3 drives (2 data + 1 parity) > raidz2 is similar to raid6 in that it is double-parity, and requires > a minimum of 4 drives (2 data + 2 parity)For any kind of RAID-Z as an absolute minimum you need one drive for data and 1, 2 or 3 drives for parity.> IOW, a raidz2 vdev made up of 3 drives will always be running in > degraded mode (it''s missing a drive)No, it will not be degraded. You can easily check it with mkfile -n 64m /var/tmp/1 /var/tmp/2 /var/tmp/3 zpool create rz2 raidz2 /var/tmp/[123] zpool status rz2 Regards Victor> -- > Freddie Cash > fjwcash at gmail.com > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/b1e73bc4/attachment.html>
On Mar 25, 2010, at 12:10 PM, Freddie Cash wrote:> On Thu, Mar 25, 2010 at 11:47 AM, Bruno Sousa <bsousa at epinfante.com> wrote: > What do you mean by "Using fewer than 4 disks in a raidz2 defeats the purpose of raidz2, as you will always be in a degraded mode" ? Does it means that having 2 vdevs with 3 disks it won''t be redundant in the advent of a drive failure? > > raidz1 is similar to raid5 in that it is single-parity, and requires a minimum of 3 drives (2 data + 1 parity)no. raidz requires a minimum of 2 drives: data + parity> raidz2 is similar to raid6 in that it is double-parity, and requires a minimum of 4 drives (2 data + 2 parity)Similarly, raidz2 requires 3 drives: data + 2 parity> > IOW, a raidz2 vdev made up of 3 drives will always be running in degraded mode (it''s missing a drive).The definition of the degraded state is in the zpool(1m) man page: DEGRADED One or more top-level vdevs is in the degraded state because one or more component devices are offline. Suf- ficient replicas exist to continue functioning. One or more component devices is in the degraded or faulted state, but sufficient replicas exist to continue functioning. The underlying conditions are as follows: o The number of checksum errors exceeds accept- able levels and the device is degraded as an indication that something may be wrong. ZFS continues to use the device as necessary. o The number of I/O errors exceeds acceptable levels. The device could not be marked as faulted because there are insufficient replicas to continue functioning. -- richard ZFS storage and performance consulting at http://www.RichardElling.com ZFS training on deduplication, NexentaStor, and NAS performance Las Vegas, April 29-30, 2010 http://nexenta-vegas.eventbrite.com
On Thu, Mar 25, 2010 at 12:44 PM, Richard Elling <richard.elling at gmail.com>wrote:> On Mar 25, 2010, at 12:10 PM, Freddie Cash wrote: > > > On Thu, Mar 25, 2010 at 11:47 AM, Bruno Sousa <bsousa at epinfante.com> > wrote: > > What do you mean by "Using fewer than 4 disks in a raidz2 defeats the > purpose of raidz2, as you will always be in a degraded mode" ? Does it means > that having 2 vdevs with 3 disks it won''t be redundant in the advent of a > drive failure? > > > > raidz1 is similar to raid5 in that it is single-parity, and requires a > minimum of 3 drives (2 data + 1 parity) > > no. raidz requires a minimum of 2 drives: data + parity > > > raidz2 is similar to raid6 in that it is double-parity, and requires a > minimum of 4 drives (2 data + 2 parity) > > Similarly, raidz2 requires 3 drives: data + 2 parity > > Coolio. Learn something new everyday. One more way that raidz isdifferent from RAID5/6/etc. So, is it just a "standard" that hardware/software RAID setups require 3 drives for a RAID5 array? And 4 drives for RAID6? -- Freddie Cash fjwcash at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100325/7e425eb7/attachment.html>
Freddie Cash wrote:> So, is it just a "standard" that hardware/software RAID setups require 3 > drives for a RAID5 array? And 4 drives for RAID6?It''s padding on the sharp edges. See my earlier post - a 2 disk RAID5 is silly, use a mirror. A 3 disk RAID6 is silly, use a 3-way mirror. Both are "legal", but most vendors don''t let you do it b/c you really don''t want to. -- Carson
> Using fewer than 4 disks in a raidz2 defeats the purpose of raidz2, as> you will always be in a degraded mode.Freddie, are you nuts? This is false. Sure you can use raidz2 with 3 disks in it. But it does seem pointless to do that instead of a 3-way mirror. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100326/21404e43/attachment.html>
> Coolio. Learn something new everyday. One more way that raidz is > different from RAID5/6/etc.Freddie, again, you''re wrong. Yes, it''s perfectly acceptable to create either raid-5 or raidz using 2 disks. It''s not degraded, but it does seem pointless to do this instead of a mirror. Likewise, it''s perfectly acceptable to create a raid-6 or raid-dp or raidz2 using 3 disks. It''s not degraded, but seems pointless to do this instead of a 3-way mirror. Since it''s pointless, some hardware vendors may not implement it in their raid controllers. They might only give you the option of creating a mirror instead. But that doesn''t mean it''s invalid raid configuration.> So, is it just a "standard" that hardware/software RAID setups require > 3 drives for a RAID5 array? And 4 drives for RAID6?It is just "standard" not to create a silly 2-disk raid5 or raidz. But don''t use the word "require." It is common practice to create raidz2 only with 4 disks or more, but again, don''t use the word "require." Some people do in fact create these silly configurations just because they''re unfamiliar with what it all means. Take Bruno''s original post as example, and that article he referenced on sun.com. How these things get started, I''ll never know.
Just because most people are probably too lazy to click the link, I?ll paste a
phrase from that sun.com webpage below:
?Creating a single-parity RAID-Z pool is identical to creating a mirrored pool,
except that the ?raidz? or ?raidz1? keyword is used instead of ?mirror?.?
And
?zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0?
So ? Shame on you, Sun, for doing this to your poor unfortunate readers. It
would be nice if the page were a wiki, or somehow able to have feedback
submitted?
From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at
opensolaris.org] On Behalf Of Bruno Sousa
Sent: Thursday, March 25, 2010 3:28 PM
To: Freddie Cash
Cc: ZFS filesystem discussion list
Subject: Re: [zfs-discuss] RAIDZ2 configuration
Hmm...it might be completely wrong , but the idea of raidz2 vdev with 3 disks
came from the reading of http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view
.
This particular page has the following example :
zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0
# zpool status -v tank
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c2t0d0 ONLINE 0 0 0
c3t0d0 ONLINE 0 0 0
So...what am i missing here? Just a bad example in the sun documentation
regarding zfs?
Bruno
On 25-3-2010 20:10, Freddie Cash wrote:
On Thu, Mar 25, 2010 at 11:47 AM, Bruno Sousa <bsousa at epinfante.com>
wrote:
What do you mean by "Using fewer than 4 disks in a raidz2 defeats the
purpose of raidz2, as you will always be in a degraded mode" ? Does it
means that having 2 vdevs with 3 disks it won''t be redundant in the
advent of a drive failure?
raidz1 is similar to raid5 in that it is single-parity, and requires a minimum
of 3 drives (2 data + 1 parity)
raidz2 is similar to raid6 in that it is double-parity, and requires a minimum
of 4 drives (2 data + 2 parity)
IOW, a raidz2 vdev made up of 3 drives will always be running in degraded mode
(it''s missing a drive).
--
Freddie Cash
fjwcash at gmail.com
--
This message has been scanned for viruses and
dangerous content by <http://www.mailscanner.info/> MailScanner, and is
believed to be clean.
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100326/5098b023/attachment.html>
On Fri, Mar 26 at 7:29, Edward Ned Harvey wrote:> > Using fewer than 4 disks in a raidz2 defeats the purpose of raidz2, as > > you will always be in a degraded mode.> Freddie, are you nuts? This is false. > > Sure you can use raidz2 with 3 disks in it. But it does seem pointless to > do that instead of a 3-way mirror.One thing about mirrors is you can put each side of your mirror on a different controller, so that any single controller failure doesn''t cause your pool to go down. While controller failure rates are very low, using 16/24 or 14/21 drives for parity on a dataset seems crazy to me. I know disks can be unreliable, but they shouldn''t be THAT unreliable. I''d think that spending fewer drives for "hot" redundancy and then spending some of the balance on an isolated warm/cold backup solution would be more cost effective. http://blog.richardelling.com/2010/02/zfs-data-protection-comparison.html Quoting from the summary, "at some point, the system design will be dominated by common failures and not the failure of independent disks." Another thought is that if heavy seeking is more likely to lead to high temperature and/or drive failure, then reserving one or two slots for an SSD L2ARC might be a good idea. It''ll take a lot of load off of your spindles if your data set fits or mostly fits within the L2ARC. You''d need a lot of RAM to make use of a large L2ARC though, just something to keep in mind. We have a 32GB X25-E as L2ARC and though it''s never more than ~5GB full with our workloads, most every file access saturates the wire (1.0 Gb/s ethernet) once the cache has warmed up, resulting in very little IO to our spindles. --eric -- Eric D. Mudama edmudama at mail.bounceswoosh.org
On Fri, March 26, 2010 07:38, Edward Ned Harvey wrote:>> Coolio. Learn something new everyday. One more way that raidz is >> different from RAID5/6/etc. > > Freddie, again, you''re wrong. Yes, it''s perfectly acceptable to create > either raid-5 or raidz using 2 disks. It''s not degraded, but it does seem > pointless to do this instead of a mirror.I think the word you''re looking for is "possible", not "acceptable".
Hi Ned, If you look at the examples on the page that you cite, they start with single-parity RAIDZ examples and then move to double-parity RAIDZ example with supporting text, here: http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view Can you restate the problem with this page? Thanks, Cindy On 03/26/10 05:42, Edward Ned Harvey wrote:> Just because most people are probably too lazy to click the link, I?ll > paste a phrase from that sun.com webpage below: > > ?Creating a single-parity RAID-Z pool is identical to creating a > mirrored pool, except that the ?raidz? or ?raidz1? keyword is used > instead of ?mirror?.? > > And > > ?zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0? > > > > So ? Shame on you, Sun, for doing this to your poor unfortunate > readers. It would be nice if the page were a wiki, or somehow able to > have feedback submitted? > > > > > > > > *From:* zfs-discuss-bounces at opensolaris.org > [mailto:zfs-discuss-bounces at opensolaris.org] *On Behalf Of *Bruno Sousa > *Sent:* Thursday, March 25, 2010 3:28 PM > *To:* Freddie Cash > *Cc:* ZFS filesystem discussion list > *Subject:* Re: [zfs-discuss] RAIDZ2 configuration > > > > Hmm...it might be completely wrong , but the idea of raidz2 vdev with 3 > disks came from the reading of > http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view . > > This particular page has the following example : > > *zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0* > > # *zpool status -v tank* > > pool: tank > > state: ONLINE > > scrub: none requested > > config: > > > > NAME STATE READ WRITE CKSUM > > tank ONLINE 0 0 0 > > raidz2 ONLINE 0 0 0 > > c1t0d0 ONLINE 0 0 0 > > c2t0d0 ONLINE 0 0 0 > > c3t0d0 ONLINE 0 0 0 > > > > So...what am i missing here? Just a bad example in the sun documentation > regarding zfs? > > Bruno > > On 25-3-2010 20:10, Freddie Cash wrote: > > On Thu, Mar 25, 2010 at 11:47 AM, Bruno Sousa <bsousa at epinfante.com > <mailto:bsousa at epinfante.com>> wrote: > > What do you mean by "Using fewer than 4 disks in a raidz2 defeats the > purpose of raidz2, as you will always be in a degraded mode" ? Does it > means that having 2 vdevs with 3 disks it won''t be redundant in the > advent of a drive failure? > > > > raidz1 is similar to raid5 in that it is single-parity, and requires a > minimum of 3 drives (2 data + 1 parity) > > raidz2 is similar to raid6 in that it is double-parity, and requires a > minimum of 4 drives (2 data + 2 parity) > > > > IOW, a raidz2 vdev made up of 3 drives will always be running in > degraded mode (it''s missing a drive). > > > > -- > > Freddie Cash > fjwcash at gmail.com <mailto:fjwcash at gmail.com> > > -- > This message has been scanned for viruses and > dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is > believed to be clean. > > > > > > _______________________________________________ > > zfs-discuss mailing list > > zfs-discuss at opensolaris.org <mailto:zfs-discuss at opensolaris.org> > > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Hi Cindy, This all issue started when i asked opinion in this list in how should i create zpools. It seems that one of my initial ideas of creating a vdev with 3 disks in a raidz configuration seems to be a non-sense configuration. Somewhere along the way i "defended" my initial idea with the fact that the documentation from Sun has as an example such configuration as seen here : *zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0* at http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view So if by concept the idea of having a vdev with 3 disks within a raidz configuration is a bad one, the oficial Sun documentation should not have such example. However if people made such example in Sun documentation, perhaps this all idea is not that bad at all.. Can you provide anything on this subject? Thanks, Bruno On 31-3-2010 23:49, Cindy Swearingen wrote:> Hi Ned, > > If you look at the examples on the page that you cite, they start > with single-parity RAIDZ examples and then move to double-parity RAIDZ > example with supporting text, here: > > http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view > > Can you restate the problem with this page? > > Thanks, > > Cindy > > > On 03/26/10 05:42, Edward Ned Harvey wrote: >> Just because most people are probably too lazy to click the link, >> I?ll paste a phrase from that sun.com webpage below: >> >> ?Creating a single-parity RAID-Z pool is identical to creating a >> mirrored pool, except that the ?raidz? or ?raidz1? keyword is used >> instead of ?mirror?.? >> >> And >> >> ?zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0? >> >> >> >> So ? Shame on you, Sun, for doing this to your poor unfortunate >> readers. It would be nice if the page were a wiki, or somehow able >> to have feedback submitted? >> >> >> >> >> >> >> >> *From:* zfs-discuss-bounces at opensolaris.org >> [mailto:zfs-discuss-bounces at opensolaris.org] *On Behalf Of *Bruno Sousa >> *Sent:* Thursday, March 25, 2010 3:28 PM >> *To:* Freddie Cash >> *Cc:* ZFS filesystem discussion list >> *Subject:* Re: [zfs-discuss] RAIDZ2 configuration >> >> >> >> Hmm...it might be completely wrong , but the idea of raidz2 vdev with >> 3 disks came from the reading of >> http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view . >> >> This particular page has the following example : >> >> *zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0* >> >> # *zpool status -v tank* >> >> pool: tank >> >> state: ONLINE >> >> scrub: none requested >> >> config: >> >> >> >> NAME STATE READ WRITE CKSUM >> >> tank ONLINE 0 0 0 >> >> raidz2 ONLINE 0 0 0 >> >> c1t0d0 ONLINE 0 0 0 >> >> c2t0d0 ONLINE 0 0 0 >> >> c3t0d0 ONLINE 0 0 0 >> >> >> >> So...what am i missing here? Just a bad example in the sun >> documentation regarding zfs? >> >> Bruno >> >> On 25-3-2010 20:10, Freddie Cash wrote: >> >> On Thu, Mar 25, 2010 at 11:47 AM, Bruno Sousa <bsousa at epinfante.com >> <mailto:bsousa at epinfante.com>> wrote: >> >> What do you mean by "Using fewer than 4 disks in a raidz2 defeats the >> purpose of raidz2, as you will always be in a degraded mode" ? Does >> it means that having 2 vdevs with 3 disks it won''t be redundant in >> the advent of a drive failure? >> >> >> >> raidz1 is similar to raid5 in that it is single-parity, and requires >> a minimum of 3 drives (2 data + 1 parity) >> >> raidz2 is similar to raid6 in that it is double-parity, and requires >> a minimum of 4 drives (2 data + 2 parity) >> >> >> >> IOW, a raidz2 vdev made up of 3 drives will always be running in >> degraded mode (it''s missing a drive). >> >> >> >> -- >> >> Freddie Cash >> fjwcash at gmail.com <mailto:fjwcash at gmail.com> >> >> -- >> This message has been scanned for viruses and >> dangerous content by *MailScanner* <http://www.mailscanner.info/>, >> and is >> believed to be clean. >> >> >> >> >> >> _______________________________________________ >> >> zfs-discuss mailing list >> >> zfs-discuss at opensolaris.org <mailto:zfs-discuss at opensolaris.org> >> >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> >> >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3656 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100401/9fc21ea3/attachment.bin>
Hi Bruno, I agree that the raidz2 example on this page is weak and I will provide a better one. ZFS is very flexible and can be configured many different ways. If someone new to ZFS wants to take 3 old (but reliable) disks and make a raidz2 configuration for testing, we would not consider this is a nonsensical idea. If I had only 3 disks, I would create a mirrored configuration of two disks and keep one as a spare. Thanks, Cindy On 03/31/10 16:02, Bruno Sousa wrote:> Hi Cindy, > > This all issue started when i asked opinion in this list in how should i > create zpools. It seems that one of my initial ideas of creating a vdev > with 3 disks in a raidz configuration seems to be a non-sense configuration. > Somewhere along the way i "defended" my initial idea with the fact that > the documentation from Sun has as an example such configuration as seen > here : > > > *zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0* at > http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view > > So if by concept the idea of having a vdev with 3 disks within a raidz > configuration is a bad one, the oficial Sun documentation should not > have such example. However if people made such example in Sun > documentation, perhaps this all idea is not that bad at all.. > > Can you provide anything on this subject? > > Thanks, > Bruno > > > > > On 31-3-2010 23:49, Cindy Swearingen wrote: >> Hi Ned, >> >> If you look at the examples on the page that you cite, they start >> with single-parity RAIDZ examples and then move to double-parity RAIDZ >> example with supporting text, here: >> >> http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view >> >> Can you restate the problem with this page? >> >> Thanks, >> >> Cindy >> >> >> On 03/26/10 05:42, Edward Ned Harvey wrote: >>> Just because most people are probably too lazy to click the link, >>> I?ll paste a phrase from that sun.com webpage below: >>> >>> ?Creating a single-parity RAID-Z pool is identical to creating a >>> mirrored pool, except that the ?raidz? or ?raidz1? keyword is used >>> instead of ?mirror?.? >>> >>> And >>> >>> ?zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0? >>> >>> >>> >>> So ? Shame on you, Sun, for doing this to your poor unfortunate >>> readers. It would be nice if the page were a wiki, or somehow able >>> to have feedback submitted? >>> >>> >>> >>> >>> >>> >>> >>> *From:* zfs-discuss-bounces at opensolaris.org >>> [mailto:zfs-discuss-bounces at opensolaris.org] *On Behalf Of *Bruno Sousa >>> *Sent:* Thursday, March 25, 2010 3:28 PM >>> *To:* Freddie Cash >>> *Cc:* ZFS filesystem discussion list >>> *Subject:* Re: [zfs-discuss] RAIDZ2 configuration >>> >>> >>> >>> Hmm...it might be completely wrong , but the idea of raidz2 vdev with >>> 3 disks came from the reading of >>> http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view . >>> >>> This particular page has the following example : >>> >>> *zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0* >>> >>> # *zpool status -v tank* >>> >>> pool: tank >>> >>> state: ONLINE >>> >>> scrub: none requested >>> >>> config: >>> >>> >>> >>> NAME STATE READ WRITE CKSUM >>> >>> tank ONLINE 0 0 0 >>> >>> raidz2 ONLINE 0 0 0 >>> >>> c1t0d0 ONLINE 0 0 0 >>> >>> c2t0d0 ONLINE 0 0 0 >>> >>> c3t0d0 ONLINE 0 0 0 >>> >>> >>> >>> So...what am i missing here? Just a bad example in the sun >>> documentation regarding zfs? >>> >>> Bruno >>> >>> On 25-3-2010 20:10, Freddie Cash wrote: >>> >>> On Thu, Mar 25, 2010 at 11:47 AM, Bruno Sousa <bsousa at epinfante.com >>> <mailto:bsousa at epinfante.com>> wrote: >>> >>> What do you mean by "Using fewer than 4 disks in a raidz2 defeats the >>> purpose of raidz2, as you will always be in a degraded mode" ? Does >>> it means that having 2 vdevs with 3 disks it won''t be redundant in >>> the advent of a drive failure? >>> >>> >>> >>> raidz1 is similar to raid5 in that it is single-parity, and requires >>> a minimum of 3 drives (2 data + 1 parity) >>> >>> raidz2 is similar to raid6 in that it is double-parity, and requires >>> a minimum of 4 drives (2 data + 2 parity) >>> >>> >>> >>> IOW, a raidz2 vdev made up of 3 drives will always be running in >>> degraded mode (it''s missing a drive). >>> >>> >>> >>> -- >>> >>> Freddie Cash >>> fjwcash at gmail.com <mailto:fjwcash at gmail.com> >>> >>> -- >>> This message has been scanned for viruses and >>> dangerous content by *MailScanner* <http://www.mailscanner.info/>, >>> and is >>> believed to be clean. >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> zfs-discuss mailing list >>> >>> zfs-discuss at opensolaris.org <mailto:zfs-discuss at opensolaris.org> >>> >>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>> >>> >>> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> zfs-discuss mailing list >>> zfs-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > >
Hi Bruno, I agree that the raidz2 example on this page is weak and I will provide a better one. ZFS is very flexible and can be configured many different ways. If someone new to ZFS wants to take 3 old (but reliable) disks and make a raidz2 configuration for testing, we would not consider this is a nonsensical idea. You can then apply what you learn about ZFS space allocation and redundancy to a new configuration. If I had only 3 disks, I would create a mirrored configuration of two disks and keep one as a spare. Thanks, Cindy On 03/31/10 16:02, Bruno Sousa wrote:> Hi Cindy, > > This all issue started when i asked opinion in this list in how should i > create zpools. It seems that one of my initial ideas of creating a vdev > with 3 disks in a raidz configuration seems to be a non-sense configuration. > Somewhere along the way i "defended" my initial idea with the fact that > the documentation from Sun has as an example such configuration as seen > here : > > > *zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0* at > http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view > > So if by concept the idea of having a vdev with 3 disks within a raidz > configuration is a bad one, the oficial Sun documentation should not > have such example. However if people made such example in Sun > documentation, perhaps this all idea is not that bad at all.. > > Can you provide anything on this subject? > > Thanks, > Bruno > > > > > On 31-3-2010 23:49, Cindy Swearingen wrote: >> Hi Ned, >> >> If you look at the examples on the page that you cite, they start >> with single-parity RAIDZ examples and then move to double-parity RAIDZ >> example with supporting text, here: >> >> http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view >> >> Can you restate the problem with this page? >> >> Thanks, >> >> Cindy >> >> >> On 03/26/10 05:42, Edward Ned Harvey wrote: >>> Just because most people are probably too lazy to click the link, >>> I?ll paste a phrase from that sun.com webpage below: >>> >>> ?Creating a single-parity RAID-Z pool is identical to creating a >>> mirrored pool, except that the ?raidz? or ?raidz1? keyword is used >>> instead of ?mirror?.? >>> >>> And >>> >>> ?zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0? >>> >>> >>> >>> So ? Shame on you, Sun, for doing this to your poor unfortunate >>> readers. It would be nice if the page were a wiki, or somehow able >>> to have feedback submitted? >>> >>> >>> >>> >>> >>> >>> >>> *From:* zfs-discuss-bounces at opensolaris.org >>> [mailto:zfs-discuss-bounces at opensolaris.org] *On Behalf Of *Bruno Sousa >>> *Sent:* Thursday, March 25, 2010 3:28 PM >>> *To:* Freddie Cash >>> *Cc:* ZFS filesystem discussion list >>> *Subject:* Re: [zfs-discuss] RAIDZ2 configuration >>> >>> >>> >>> Hmm...it might be completely wrong , but the idea of raidz2 vdev with >>> 3 disks came from the reading of >>> http://docs.sun.com/app/docs/doc/819-5461/gcvjg?a=view . >>> >>> This particular page has the following example : >>> >>> *zpool create tank raidz2 c1t0d0 c2t0d0 c3t0d0* >>> >>> # *zpool status -v tank* >>> >>> pool: tank >>> >>> state: ONLINE >>> >>> scrub: none requested >>> >>> config: >>> >>> >>> >>> NAME STATE READ WRITE CKSUM >>> >>> tank ONLINE 0 0 0 >>> >>> raidz2 ONLINE 0 0 0 >>> >>> c1t0d0 ONLINE 0 0 0 >>> >>> c2t0d0 ONLINE 0 0 0 >>> >>> c3t0d0 ONLINE 0 0 0 >>> >>> >>> >>> So...what am i missing here? Just a bad example in the sun >>> documentation regarding zfs? >>> >>> Bruno >>> >>> On 25-3-2010 20:10, Freddie Cash wrote: >>> >>> On Thu, Mar 25, 2010 at 11:47 AM, Bruno Sousa <bsousa at epinfante.com >>> <mailto:bsousa at epinfante.com>> wrote: >>> >>> What do you mean by "Using fewer than 4 disks in a raidz2 defeats the >>> purpose of raidz2, as you will always be in a degraded mode" ? Does >>> it means that having 2 vdevs with 3 disks it won''t be redundant in >>> the advent of a drive failure? >>> >>> >>> >>> raidz1 is similar to raid5 in that it is single-parity, and requires >>> a minimum of 3 drives (2 data + 1 parity) >>> >>> raidz2 is similar to raid6 in that it is double-parity, and requires >>> a minimum of 4 drives (2 data + 2 parity) >>> >>> >>> >>> IOW, a raidz2 vdev made up of 3 drives will always be running in >>> degraded mode (it''s missing a drive). >>> >>> >>> >>> -- >>> >>> Freddie Cash >>> fjwcash at gmail.com <mailto:fjwcash at gmail.com> >>> >>> -- >>> This message has been scanned for viruses and >>> dangerous content by *MailScanner* <http://www.mailscanner.info/>, >>> and is >>> believed to be clean. >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> zfs-discuss mailing list >>> >>> zfs-discuss at opensolaris.org <mailto:zfs-discuss at opensolaris.org> >>> >>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >>> >>> >>> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> zfs-discuss mailing list >>> zfs-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > >
Cindy Swearingen wrote:> If someone new to ZFS wants to take 3 old (but reliable) disks and make > a raidz2 configuration for testing, we would not consider this is a > nonsensical idea. You can then apply what you learn about ZFS space > allocation and redundancy to a new configuration."Nonsensical" may be a bit strong, but I can see no possible use case where a 3 disk raidz2 isn''t better served by a 3-way mirror. Of course many things that are terrible ideas in production become useful learning experiences ;-) -- Carson
On Thu, Apr 1, 2010 at 11:46 AM, Carson Gaspar <carson at taltos.org> wrote:> "Nonsensical" may be a bit strong, but I can see no possible use case where > a 3 disk raidz2 isn''t better served by a 3-way mirror. >Once bp_rewrite is done, you''ll be able add disks to the raidz2. I suppose that''s one reason? -B -- Brandon High : bhigh at freaks.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100401/e0bf4f03/attachment.html>
Brandon High wrote:> On Thu, Apr 1, 2010 at 11:46 AM, Carson Gaspar <carson at taltos.org > <mailto:carson at taltos.org>> wrote: > > "Nonsensical" may be a bit strong, but I can see no possible use > case where a 3 disk raidz2 isn''t better served by a 3-way mirror. > > > Once bp_rewrite is done, you''ll be able add disks to the raidz2. I > suppose that''s one reason?If you can expand a raidz2, you should be able to convert a mirror into a raidz2. Actually, you can do that now (with some trickery): - zpool detach oldpool exmirror3 - Add your new disk(s) to the system - mkfile /blah/fake1; mkfile /blah/fake2 - zpool create newpool raidz2 exmirror3 newdisk1 [newdisk2...] fake1 fake2 - zpool offline newpool fake1 fake2 - copy oldpool to newpool with zfs send/recv - zpool detach oldpool exmirror2 - zpool replace newpool fake2 exmirror2 - zpool destroy oldpool - zpool replace newpool fake1 exmirror1 3-way mirror becomes 4 or more disk raidz2 while retaining redundancy except for the brief period when you are running with a single disk old pool and a resilvering raidz2, and even then you have 2 copies of your data. Sneaky, eh? -- Carson