I am wanting to build a server with 16 - 1TB drives with 2 ? 8 drive RAID Z2 arrays striped together. However, I would like the capability of adding additional stripes of 2TB drives in the future. Will this be a problem? I thought I read it is best to keep the stripes the same width and was planning to do that, but I was wondering about using drives of different sizes. These drives would all be in a single pool. -- Terry Hull Network Resource Group, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100810/529ce953/attachment.html>
I am wanting to build a server with 16 - 1TB drives with 2 ? 8 drive RAID Z2 arrays striped together. However, I would like the capability of adding additional stripes of 2TB drives in the future. Will this be a problem? I thought I read it is best to keep the stripes the same width and was planning to do that, but I was wondering about using drives of different sizes. These drives would all be in a single pool. -- Terry Hull Network Resource Group, Inc. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100810/041da37d/attachment.html>
On 08/10/10 06:21 PM, Terry Hull wrote:> I am wanting to build a server with 16 - 1TB drives with 2 ? 8 drive > RAID Z2 arrays striped together. However, I would like the capability > of adding additional stripes of 2TB drives in the future. Will this be > a problem? I thought I read it is best to keep the stripes the same > width and was planning to do that, but I was wondering about using > drives of different sizes. These drives would all be in a single pool. >It would work, but you run the risk of the smaller drives becoming full and all new writes doing to the bigger vdev. So while usable, performance would suffer. One option would be to add 2TB drives as 5 drive raidz3 vdevs. That way your vdevs would be approximately the same size and you would have the optimum redundancy for the 2TB drives. -- Ian.
On 10 Aug 2010, at 08:49, Ian Collins <ian at ianshome.com> wrote:> On 08/10/10 06:21 PM, Terry Hull wrote: >> I am wanting to build a server with 16 - 1TB drives with 2 ? 8 dri >> ve RAID Z2 arrays striped together. However, I would like the capa >> bility of adding additional stripes of 2TB drives in the future. W >> ill this be a problem? I thought I read it is best to keep the str >> ipes the same width and was planning to do that, but I was wonderi >> ng about using drives of different sizes. These drives would all b >> e in a single pool. >> > It would work, but you run the risk of the smaller drives becoming > full and all new writes doing to the bigger vdev. So while usable, > performance would suffer.Almost by definition, the 1TB drives are likely to be getting full when the new drives are added (presumably because of running out of space). Performance can only be said to suffer relative to a new pool built entirely with drives of the same size. Even if he added 8x 2TB drives in a RAIDZ3 config it is hard to predict what the performance gap will be (on the one hand: RAIDZ3 vs RAIDZ2, on the other: an empty group vs an almost full, presumably fragmented, group).> One option would be to add 2TB drives as 5 drive raidz3 vdevs. That > way your vdevs would be approximately the same size and you would > have the optimum redundancy for the 2TB drives.I think you meant 6, but I don''t see a good reason for matching the group sizes. I''m for RAIDZ3, but I don''t see much logic in mixing groups of 6+2 x 1TB and 3+3 x 2TB in the same pool (in one group I appear to care most about maximising space, in the other I''m maximising availability) The other issue is that of hot spares. In a pool of mixed size drives you either waste array slots (by having spares of different sizes) or plan to have unavailable space when small drives are replaced by large ones.>
Phil Harman wrote:> On 10 Aug 2010, at 08:49, Ian Collins <ian at ianshome.com> wrote: > >> On 08/10/10 06:21 PM, Terry Hull wrote: >>> I am wanting to build a server with 16 - 1TB drives with 2 ? 8 drive >>> RAID Z2 arrays striped together. However, I would like the >>> capability of adding additional stripes of 2TB drives in the future. >>> Will this be a problem? I thought I read it is best to keep the >>> stripes the same width and was planning to do that, but I was >>> wondering about using drives of different sizes. These drives would >>> all be in a single pool. >>> >> It would work, but you run the risk of the smaller drives becoming >> full and all new writes doing to the bigger vdev. So while usable, >> performance would suffer. > > Almost by definition, the 1TB drives are likely to be getting full > when the new drives are added (presumably because of running out of > space). > > Performance can only be said to suffer relative to a new pool built > entirely with drives of the same size. Even if he added 8x 2TB drives > in a RAIDZ3 config it is hard to predict what the performance gap will > be (on the one hand: RAIDZ3 vs RAIDZ2, on the other: an empty group vs > an almost full, presumably fragmented, group). > >> One option would be to add 2TB drives as 5 drive raidz3 vdevs. That >> way your vdevs would be approximately the same size and you would >> have the optimum redundancy for the 2TB drives. > > I think you meant 6, but I don''t see a good reason for matching the > group sizes. I''m for RAIDZ3, but I don''t see much logic in mixing > groups of 6+2 x 1TB and 3+3 x 2TB in the same pool (in one group I > appear to care most about maximising space, in the other I''m > maximising availability)Another option - use the new 2TB drives to swap out the existing 1TB drives. If you can find another use for the swapped out drives, this works well, and avoids ending up with sprawling lower capacity drives as your pool grows in size. This is what I do at home. The freed-up drives get used in other systems and for off-site backups. Over the last 4 years, I''ve upgraded from 1/4TB, to 1/2TB, and now on 1TB drives. -- Andrew Gabriel
On 08/10/10 09:12 PM, Andrew Gabriel wrote:> Phil Harman wrote: >> On 10 Aug 2010, at 08:49, Ian Collins <ian at ianshome.com> wrote: >>> On 08/10/10 06:21 PM, Terry Hull wrote: >>>> I am wanting to build a server with 16 - 1TB drives with 2 ? 8 >>>> drive RAID Z2 arrays striped together. However, I would like the >>>> capability of adding additional stripes of 2TB drives in the >>>> future. Will this be a problem? I thought I read it is best to keep >>>> the stripes the same width and was planning to do that, but I was >>>> wondering about using drives of different sizes. These drives would >>>> all be in a single pool. >>>> >>> It would work, but you run the risk of the smaller drives becoming >>> full and all new writes doing to the bigger vdev. So while usable, >>> performance would suffer. >> >> Almost by definition, the 1TB drives are likely to be getting full >> when the new drives are added (presumably because of running out of >> space). >> >> Performance can only be said to suffer relative to a new pool built >> entirely with drives of the same size. Even if he added 8x 2TB drives >> in a RAIDZ3 config it is hard to predict what the performance gap >> will be (on the one hand: RAIDZ3 vs RAIDZ2, on the other: an empty >> group vs an almost full, presumably fragmented, group). >> >>> One option would be to add 2TB drives as 5 drive raidz3 vdevs. That >>> way your vdevs would be approximately the same size and you would >>> have the optimum redundancy for the 2TB drives. >> >> I think you meant 6, but I don''t see a good reason for matching the >> group sizes. I''m for RAIDZ3, but I don''t see much logic in mixing >> groups of 6+2 x 1TB and 3+3 x 2TB in the same pool (in one group I >> appear to care most about maximising space, in the other I''m >> maximising availability) > > Another option - use the new 2TB drives to swap out the existing 1TB > drives. > If you can find another use for the swapped out drives, this works > well, and avoids ending up with sprawling lower capacity drives as > your pool grows in size. This is what I do at home. The freed-up > drives get used in other systems and for off-site backups. Over the > last 4 years, I''ve upgraded from 1/4TB, to 1/2TB, and now on 1TB drives. >I have been doing the same. The reason I mentioned performance (and I did mean 6 drives!) is in order to get some space on a budget I replaced one mirror in a stripe with bigger drives. The others soon became nearly full and most of the IO went to the bigger pair, so I lost nearly all the benefit of the stripe. I have also grown stripes and seen similar issues and I had to remove and replace large chunks of data to even things out. I really think mixing vdev sizes is a bad idea. -- Ian.
On 10 Aug 2010, at 10:22, Ian Collins <ian at ianshome.com> wrote:> On 08/10/10 09:12 PM, Andrew Gabriel wrote: >> Phil Harman wrote: >>> On 10 Aug 2010, at 08:49, Ian Collins <ian at ianshome.com> wrote: >>>> On 08/10/10 06:21 PM, Terry Hull wrote: >>>>> I am wanting to build a server with 16 - 1TB drives with 2 ? 8 drive RAID Z2 arrays striped together. However, I would like the capability of adding additional stripes of 2TB drives in the future. Will this be a problem? I thought I read it is best to keep the stripes the same width and was planning to do that, but I was wondering about using drives of different sizes. These drives would all be in a single pool. >>>>> >>>> It would work, but you run the risk of the smaller drives becoming full and all new writes doing to the bigger vdev. So while usable, performance would suffer. >>> >>> Almost by definition, the 1TB drives are likely to be getting full when the new drives are added (presumably because of running out of space). >>> >>> Performance can only be said to suffer relative to a new pool built entirely with drives of the same size. Even if he added 8x 2TB drives in a RAIDZ3 config it is hard to predict what the performance gap will be (on the one hand: RAIDZ3 vs RAIDZ2, on the other: an empty group vs an almost full, presumably fragmented, group). >>> >>>> One option would be to add 2TB drives as 5 drive raidz3 vdevs. That way your vdevs would be approximately the same size and you would have the optimum redundancy for the 2TB drives. >>> >>> I think you meant 6, but I don''t see a good reason for matching the group sizes. I''m for RAIDZ3, but I don''t see much logic in mixing groups of 6+2 x 1TB and 3+3 x 2TB in the same pool (in one group I appear to care most about maximising space, in the other I''m maximising availability) >> >> Another option - use the new 2TB drives to swap out the existing 1TB drives. >> If you can find another use for the swapped out drives, this works well, and avoids ending up with sprawling lower capacity drives as your pool grows in size. This is what I do at home. The freed-up drives get used in other systems and for off-site backups. Over the last 4 years, I''ve upgraded from 1/4TB, to 1/2TB, and now on 1TB drives. >> > I have been doing the same. > > The reason I mentioned performance (and I did mean 6 drives!) is in order to get some space on a budget I replaced one mirror in a stripe with bigger drives. The others soon became nearly full and most of the IO went to the bigger pair, so I lost nearly all the benefit of the stripe. I have also grown stripes and seen similar issues and I had to remove and replace large chunks of data to even things out. > > I really think mixing vdev sizes is a bad idea.I''d agree if this was a new pool, but this question was about expanding an existing pool (which is nearly full and where the performance is presumably acceptable). Adding another vdev, whatever its size, is a simple zero downtime option for growing the pool (adding another pool would fragment the name space). With a similar number of spindles in a similar RAID configuration, performance is unlikely to get worse, indeed (as already noted) it is likely to get better until the new vdev fills up. Many systems only need to be good enough, not optimum. The best is often the enemy of the good. Anyone using RAIDZn is cost conscious to some degree (or why not just go for a hige stripe of 4-way mirrored SSDs and be done with it?)
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Terry Hull > > I am wanting to build a server with 16 - 1TB drives with 2 ? 8 drive > RAID Z2 arrays striped together. ??However, I would like the capability > of adding additional stripes of 2TB drives in the future. ?Will this be > a problem? ??I thought I read it is best to keep the stripes the same > width and was planning to do that, but I was wondering about using > drives of different sizes. ?These drives would all be in a single pool.There is no problem. Even if you wanted to add all sorts of randomly sized & shaped vdevs in the future, it would work. The reason they recommend sticking with one type of configuration is because of performance characteristics. That''s not necessarily to say it performs better with only a single type, it''s more "you don''t know what to expect" if you have a raidz striped with a mirror and a raidz2, etc.
> From: Phil Harman <phil.harman at gmail.com> > Date: Tue, 10 Aug 2010 09:24:52 +0100 > To: Ian Collins <ian at ianshome.com> > Cc: Terry Hull <tah at nrg-inc.com>, "zfs-discuss at opensolaris.org" > <zfs-discuss at opensolaris.org> > Subject: Re: [zfs-discuss] RAID Z stripes > > On 10 Aug 2010, at 08:49, Ian Collins <ian at ianshome.com> wrote: > >> On 08/10/10 06:21 PM, Terry Hull wrote: >>> I am wanting to build a server with 16 - 1TB drives with 2 ? 8 dri >>> ve RAID Z2 arrays striped together. However, I would like the capa >>> bility of adding additional stripes of 2TB drives in the future. W >>> ill this be a problem? I thought I read it is best to keep the str >>> ipes the same width and was planning to do that, but I was wonderi >>> ng about using drives of different sizes. These drives would all b >>> e in a single pool. >>> >> It would work, but you run the risk of the smaller drives becoming >> full and all new writes doing to the bigger vdev. So while usable, >> performance would suffer. > > Almost by definition, the 1TB drives are likely to be getting full > when the new drives are added (presumably because of running out of > space). > > Performance can only be said to suffer relative to a new pool built > entirely with drives of the same size. Even if he added 8x 2TB drives > in a RAIDZ3 config it is hard to predict what the performance gap will > be (on the one hand: RAIDZ3 vs RAIDZ2, on the other: an empty group vs > an almost full, presumably fragmented, group). > >> One option would be to add 2TB drives as 5 drive raidz3 vdevs. That >> way your vdevs would be approximately the same size and you would >> have the optimum redundancy for the 2TB drives. > > I think you meant 6, but I don''t see a good reason for matching the > group sizes. I''m for RAIDZ3, but I don''t see much logic in mixing > groups of 6+2 x 1TB and 3+3 x 2TB in the same pool (in one group I > appear to care most about maximising space, in the other I''m > maximising availability) > > The other issue is that of hot spares. In a pool of mixed size drives > you either waste array slots (by having spares of different sizes) or > plan to have unavailable space when small drives are replaced by large > ones.So do I understand correctly that really the "Right" thing to do is to build a pool not only with a consistent strip width, but also to build it with drives on only one size? It also sounds like from a practical point of view that building the pool full-sized is the best policy so that the data can be spread relatively uniformly across all the drives from the very beginning. In my case, I think what I will do is to start with the 16 drives in a single pool and when I need more space, I''ll create a new pool and manually move the some of the existing data to the new pool to spread the IO load. The other issue here seems to be RAIDZ2 vs RAIDZ3. I assume there is not a significant performance difference between the two for most loads, but rather I choose between them based on how badly I want the array to stay intact. - Terry
On 08/10/10 10:09 PM, Phil Harman wrote:> On 10 Aug 2010, at 10:22, Ian Collins<ian at ianshome.com> wrote: > > >> On 08/10/10 09:12 PM, Andrew Gabriel wrote: >>> Another option - use the new 2TB drives to swap out the existing 1TB drives. >>> If you can find another use for the swapped out drives, this works well, and avoids ending up with sprawling lower capacity drives as your pool grows in size. This is what I do at home. The freed-up drives get used in other systems and for off-site backups. Over the last 4 years, I''ve upgraded from 1/4TB, to 1/2TB, and now on 1TB drives. >>> >>> >> I have been doing the same. >> >> The reason I mentioned performance (and I did mean 6 drives!) is in order to get some space on a budget I replaced one mirror in a stripe with bigger drives. The others soon became nearly full and most of the IO went to the bigger pair, so I lost nearly all the benefit of the stripe. I have also grown stripes and seen similar issues and I had to remove and replace large chunks of data to even things out. >> >> I really think mixing vdev sizes is a bad idea. >> > I''d agree if this was a new pool, but this question was about expanding an existing pool (which is nearly full and where the performance is presumably acceptable). > > Adding another vdev, whatever its size, is a simple zero downtime option for growing the pool (adding another pool would fragment the name space). With a similar number of spindles in a similar RAID configuration, performance is unlikely to get worse, indeed (as already noted) it is likely to get better until the new vdev fills up. > >The best option for growing a pool is often swapping out the drives for larger ones, which is also a zero down time option.> Many systems only need to be good enough, not optimum. The best is often the enemy of the good. Anyone using RAIDZn is cost conscious to some degree (or why not just go for a hige stripe of 4-way mirrored SSDs and be done with it?) >That depends on the situation. If a particular topology was chosen to give a capacity/performance trade off, degrading one or another may not be acceptable. -- Ian.
On 08/11/10 05:16 AM, Terry Hull wrote:> So do I understand correctly that really the "Right" thing to do is to build > a pool not only with a consistent strip width, but also to build it with > drives on only one size? It also sounds like from a practical point of > view that building the pool full-sized is the best policy so that the data > can be spread relatively uniformly across all the drives from the very > beginning. In my case, I think what I will do is to start with the 16 > drives in a single pool and when I need more space, I''ll create a new pool > and manually move the some of the existing data to the new pool to spread > the IO load. > >That is what I have done when Thumpers fill up!> The other issue here seems to be RAIDZ2 vs RAIDZ3. I assume there is not a > significant performance difference between the two for most loads, but > rather I choose between them based on how badly I want the array to stay > intact. > >The real issue is how long large capacity drives take to resilver and is the risk of loosing a second drive during that window high enough to cause concern. In a lot of situations with 2TB drives, it is. -- Ian.
Ian Collins wrote:> On 08/11/10 05:16 AM, Terry Hull wrote: >> So do I understand correctly that really the "Right" thing to do is >> to build >> a pool not only with a consistent strip width, but also to build it with >> drives on only one size? It also sounds like from a practical point of >> view that building the pool full-sized is the best policy so that the >> data >> can be spread relatively uniformly across all the drives from the very >> beginning. In my case, I think what I will do is to start with the 16 >> drives in a single pool and when I need more space, I''ll create a new >> pool >> and manually move the some of the existing data to the new pool to >> spread >> the IO load. >> >> > That is what I have done when Thumpers fill up! > >> The other issue here seems to be RAIDZ2 vs RAIDZ3. I assume there is >> not a >> significant performance difference between the two for most loads, but >> rather I choose between them based on how badly I want the array to stay >> intact. >> >> > The real issue is how long large capacity drives take to resilver and > is the risk of loosing a second drive during that window high enough > to cause concern. In a lot of situations with 2TB drives, it is. >Of course your redundancy requirements may be such that you are okay with taking that risk because you have a complete backup elsewhere that you can restore from in case a second drive is lost during that recovery window. Evaluate your requirements and see if it makes more sense to dedicate more drives to make a RAIDZ3 vs RAIDZ2 (or by the same argument RAIDZ2 vs RAIDZ1), than it does to dedicate those additional drives to backup. Remember that redundancy provided by RAID is not a replacement for backup. (I get the feeling that a little bit of redundancy (RAIDZ1 or RAIDZ2) with a little bit of backup (stripe or RAIDZ1) is the best compromise overall, with forces such these pushing things one way or the other: outage tolerance, time to recover from backup, and the amount of pain induced by a complete system failure in case a non-backed up RAIDZ3 falls apart.) Running a RAIDZ3 with no backup seems like one is saying "I really really want to protect against individual drive failures, but if lightning strikes or the water main upstairs breaks I''ll give up." Running a main pool with a non-redundant stripe and two backups says "I can tolerate losing my day-to-day changes to the pool since the last backup was run, but I really don''t want to lose the bulk of my stuff which the backup protects." Running a main pool with a little bit of redundancy (e.g. RAIDZ1) and a backup with redundancy says "I don''t like downtime nor data loss of my day-to-day changes. I''m going to guard against single drive failures which is the most likely case, but resort to my more time-intensive to recover backups if I hit a bad streak and get two+ simultaneous drive failures during a long resilver. My backups have redundancy as well because if I need to fall back upon them, then it is because I really really need them." E.g.with 24 x 1TB drives, some options include: 3 x 3x1TB RAIDZ1, 6TB usable for data, 9 drives + 2x (separate pools, not striped) 7x1TB RAIDZ1 6TB usable for backup, 14 drives + 1 hot spare. Low strength main pool. Two complete low strength backups. Hot spare. 3 x 4x1TB RAIDZ2, 6TB usable for data, 12 drives + 2x (separate pools, not striped) 6x1TB stripe (no redundancy) 6TB usable for backup, 12 drives. Strong main pool. Two complete but fragile backups. 3 x 4x1TB RAIDZ1, 9TB usable for data, 12 drives + 1x 12x1TB RAIDZ3 9TB usable for backup, 12 drives. Low strength main pool. Strong backup. 3 x 5x1TB RAIDZ2, 9TB usable for data ,15 drives + 1x 9x1TB stripe (no redundancy) 9TB usable for backup, 9 drives. Strong main pool. Fragile backup. 3 x 6x1TB RAIDZ3, 9TB usable for data ,18 drives + 1x 6x1TB stripe (no redundancy) 6TB usable for backup, 6 drives. Very strong main pool. Fragile, possibly incomplete backup (depending on compression and whether part of the directory structure is not backed up) 4 x 3x1TB RAIDZ1, 8TB usable for data, 12 drives + 1x 12x1TB RAIDZ3 9TB usable for backup, 12 drives + 0 hot spares 4 x 3x1TB RAIDZ1, 8TB usable for data, 12 drives + 1x 10x1TB RAIDZ2 8TB usable for backup, 10 drives + 2 hot spares. Low strength main pool. Strong backup. Hot spares. 4 x 4x1TB RAIDZ1, 12TB usable for data, 16 drives + 1x 12x1TB stripe (no redundancy) 12TB usable for backup, 8 drives + 0 hot spares. Low strength main pool. Fragile backup. 4 x 4x1TB RAIDZ2, 8TB usable for data ,16 drives + 1x 8x1TB stripe (no redundancy) 8TB usable for backup, 8 drives + 0 hot spares. Strong main pool. Fragile backup.