I''ve been reading through the documentation on ZFS, and was hoping I could get some clarification and make sure I''m reading everything right. I''m looking to build a NAS box, using sata drives in a double parity configuration (i.e. raidz2). This is mainly for warehousing large video files, so access speed and IOPS aren''t terribly critical. I won''t immediately need all 15 drives online, so what I would like to do is be able to resize the raid array. Is there any way to resize with ZFS? Or would the only way to do this be to make one raidz2 with 7 drives, and then when I need to expand make another raidz2 in the same pool. Obviously having two virtual devices in the pool means losing 4 drives to parity instead of just two. ie start out with zpool create tank raidz2 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c2t1d0 c2t2d0 c2t3d0 and when i''m ready to expand use zpool add tank raidz2 c2t4d0 c3t1d0 c3t2d0 c3t3d0 c3t4d0 c4t1d0 c4t2d0 c4t3d0 Any and all comments are appreciated. This message posted from opensolaris.org
Scott Roberts <scott at scottmroberts.com> writes:> I''ve been reading through the documentation on ZFS, and was hoping I > could get some clarification and make sure I''m reading everything > right. > > I''m looking to build a NAS box, using sata drives in a double parity > configuration (i.e. raidz2). This is mainly for warehousing large > video files, so access speed and IOPS aren''t terribly critical. I > won''t immediately need all 15 drives online, so what I would like to > do is be able to resize the raid array. > > Is there any way to resize with ZFS? Or would the only way to do > this be to make one raidz2 with 7 drives, and then when I need to > expand make another raidz2 in the same pool. Obviously having two > virtual devices in the pool means losing 4 drives to parity instead > of just two. > > ie start out with > zpool create tank raidz2 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c2t1d0 c2t2d0 c2t3d0 > > and when i''m ready to expand use > zpool add tank raidz2 c2t4d0 c3t1d0 c3t2d0 c3t3d0 c3t4d0 c4t1d0 c4t2d0 c4t3d0I wasn''t clear on this myself until earlier this week. I *think* I understand it now, so let me try, and people here can correct me if necessary. You can''t add drives to an existing RAIDZ. You can add drives to an existing mirror, but they don''t add space, they add reliability. What you *can* do is add additional "things" to your pool, and the space on them becomes available to all filesystems and such drawing from that pool. So yes, you can do what the second thing you describe. The reliability of the pool becomes the reliability of the least-reliable thing in it; so if you have a pool with a RAIDZ in it, and then you add a single drive, everything works and the extra space is available and is used -- and if the single drive fails, you lose data. What you''re proposing, adding a second RAIDZ, is fine, and doesn''t compromise reliability. It does, as you say, take up another whole parity disk (or two in your raidz2 case). And requires add-ons to be in units bigger than just one drive. -- David Dyer-Bennet, <mailto:dd-b at dd-b.net>, <http://www.dd-b.net/dd-b/> RKBA: <http://www.dd-b.net/carry/> Pics: <http://dd-b.lighthunters.net/> <http://www.dd-b.net/dd-b/SnapshotAlbum/> Dragaera/Steven Brust: <http://dragaera.info/>
Hi Scott, You can''t add devices to an existing raidz1 or raidz2 "stripe". And the reason you can''t is precisely because of full-stripe -- but variable width -- writes that RAID-Z uses to avoid the RAID-5 write hole. Let''s say you have a 4+2 raidz2 configuration and you''re doing a bunch of 1 sector writes: | P | P | D | P | P | D | | P | P | D | P | P | D | ... There may be several parity sectors per row so adding another column doesn''t work. To expand your pool, just add another raidz2 vdev of 7 disks. You will be using two additional parity disks, but the number of parity disks you use for a given configuration should ideally depend on the fault domain for those disks (controllers, fans, etc). Adam On Wed, Jul 12, 2006 at 12:04:58PM -0700, Scott Roberts wrote:> I''ve been reading through the documentation on ZFS, and was hoping I could get some clarification and make sure I''m reading everything right. > > I''m looking to build a NAS box, using sata drives in a double parity configuration (i.e. raidz2). This is mainly for warehousing large video files, so access speed and IOPS aren''t terribly critical. I won''t immediately need all 15 drives online, so what I would like to do is be able to resize the raid array. > > Is there any way to resize with ZFS? Or would the only way to do this be to make one raidz2 with 7 drives, and then when I need to expand make another raidz2 in the same pool. Obviously having two virtual devices in the pool means losing 4 drives to parity instead of just two. > > ie start out with > zpool create tank raidz2 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c2t1d0 c2t2d0 c2t3d0 > > and when i''m ready to expand use > zpool add tank raidz2 c2t4d0 c3t1d0 c3t2d0 c3t3d0 c3t4d0 c4t1d0 c4t2d0 c4t3d0 > > Any and all comments are appreciated. > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Adam Leventhal, Solaris Kernel Development http://blogs.sun.com/ahl
Just out of curiosity, what is the progress on allowing the addition of drives to an existing RAIDZ (whether pool or udev). Particularly in the case of udevs, the ability to add additional drives to expand a udev is really useful when adding more JBODs to an existing setup... -- Erik Trimble Java System Support Mailstop: usca14-102 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800)
> You can''t add devices to an existing raidz1 or raidz2 "stripe". And the > reason you can''t is precisely because of full-stripe -- but variable width -- > writes that RAID-Z uses to avoid the RAID-5 write hole. Let''s say you have > a 4+2 raidz2 configuration and you''re doing a bunch of 1 sector writes: > > | P | P | D | P | P | D | > | P | P | D | P | P | D | > ... > > There may be several parity sectors per row so adding another column doesn''t > work.But presumably it would be possible to use additional columns for future writes? new new |1P |1P |1D |2P |2P |2D | 0 | 0 | |3P |3P |3D |4P |4P |4D | 0 | 0 | |5P |5P |5D |6P |6P |6D |7P |7P | |7D |8P |8P |8D |9P |9P |9D | 0 | The fs blocks retain their own parity, so there''s no need to zero the disks when adding. Presumably if you did this only when the earlier disks were completely almost full, you might not have enough columns to complete writes, but before that point you should be okay. Over time, freeing existing blocks would allow more space for expanded writes. (or perhaps a background process to rewrite them). That''s just a thought. I have no idea if such a scheme would be technically feasable given the existing code. -- Darren Dunham ddunham at taos.com Senior Technical Consultant TAOS http://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. >
David Dyer-Bennet <dd-b at dd-b.net> writes:> It does, as you say, take up another whole parity disk (or two in your > raidz2 case). And requires add-ons to be in units bigger than just > one drive.I''ve seen people wondering if ZFS was a scam because the claims just seemed too good to be true. Given that ZFS *is* really great, I don''t think it would hurt to prominently advertise limitations like this one it would probably benefit credibility considerably, and it''s a real consideration for anyone who''s doing RAID-Z. -- Dave Abrahams Boost Consulting www.boost-consulting.com
On Wed, Jul 12, 2006 at 02:45:40PM -0700, Darren Dunham wrote:> > There may be several parity sectors per row so adding another column doesn''t > > work. > > But presumably it would be possible to use additional columns for future > writes?I guess that could be made to work, but then the data on the disk becomes much (much much) more difficult to interpret because you have some rows which are effectively one width and others which are another (ad infinitum). It also doesn''t really address the issue since you assume that you want to add space because the disks are getting full, but this scheme, as you mention, only applies the new width to empty rows. I''m not sure I even agree with the notion that this is a real problem (and if it is, I don''t think is easily solved). Stripe widths are a function of the expected failure rate and fault domains of the system which tend to be static in nature. A coarser solution would be to create a new pool where you zfs send/zfs recv the filesystems of the old pool. Adam -- Adam Leventhal, Solaris Kernel Development http://blogs.sun.com/ahl
On Jul 12, 2006, at 20:10, Richard Elling wrote:> G1. for 3 <= Ndisks <=5, RAID-Z2 offers the best resiliency/space. > For > Ndisks > 5, RAID-1 offers best resiliency/space. Ndisk-way RAID-1 > always wins the resiliency, availability, and MTTDL but most > people > won''t do it for Ndisks > 2.Can you give some numeric examples for one or two cases? I''m having troubling ''visualizing'' why this would be the case (especially for N > 5).
There are two questions here. 1. Can you add a redundant set of vdevs to a pool. Answer: yes. 2. What is the best way for Scott to grow his archive into his disks. The answer to this is what I discuss below. David Dyer-Bennet wrote:> Scott Roberts <scott at scottmroberts.com> writes: > >> I''ve been reading through the documentation on ZFS, and was hoping I >> could get some clarification and make sure I''m reading everything >> right. >> >> I''m looking to build a NAS box, using sata drives in a double parity >> configuration (i.e. raidz2). This is mainly for warehousing large >> video files, so access speed and IOPS aren''t terribly critical. I >> won''t immediately need all 15 drives online, so what I would like to >> do is be able to resize the raid array. >> >> Is there any way to resize with ZFS? Or would the only way to do >> this be to make one raidz2 with 7 drives, and then when I need to >> expand make another raidz2 in the same pool. Obviously having two >> virtual devices in the pool means losing 4 drives to parity instead >> of just two. >> >> ie start out with >> zpool create tank raidz2 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c2t1d0 c2t2d0 c2t3d0 >> >> and when i''m ready to expand use >> zpool add tank raidz2 c2t4d0 c3t1d0 c3t2d0 c3t3d0 c3t4d0 c4t1d0 c4t2d0 c4t3d0 > > I wasn''t clear on this myself until earlier this week. I *think* I > understand it now, so let me try, and people here can correct me if > necessary. > > You can''t add drives to an existing RAIDZ. You can add drives to an > existing mirror, but they don''t add space, they add reliability. > > What you *can* do is add additional "things" to your pool, and the > space on them becomes available to all filesystems and such drawing > from that pool. So yes, you can do what the second thing you > describe. > > The reliability of the pool becomes the reliability of the > least-reliable thing in it; so if you have a pool with a RAIDZ in it, > and then you add a single drive, everything works and the extra space > is available and is used -- and if the single drive fails, you lose > data. What you''re proposing, adding a second RAIDZ, is fine, and > doesn''t compromise reliability.Not really. Though this may help you visualize the situation, the reliability is directly related to the number of disks in the pool.> It does, as you say, take up another whole parity disk (or two in your > raidz2 case). And requires add-ons to be in units bigger than just > one drive.The data availability and mean time to data loss are the two important views you need to understand to arrive at a best RAS solution. These are, in turn, weighed against effective space and performance. Some general guidelines might be: G1. for 3 <= Ndisks <=5, RAID-Z2 offers the best resiliency/space. For Ndisks > 5, RAID-1 offers best resiliency/space. Ndisk-way RAID-1 always wins the resiliency, availability, and MTTDL but most people won''t do it for Ndisks > 2. G2. Hot spares are needed to drive to high data availability and MTTDL. G3. Shutdown on multiple failures is useful for increasing MTTDL. G4. Data availability is highly dependent on the recovery time which is highly dependent on the amount of data you have. G5. Data scrubbing intervals impact MTTDL. Scrub at least once per year. G6. Non-storage components (eg. controllers) affect availability, but not MTTDL. Specific recommendations can only make sense when more specific information on the requirements is available. -- richard
David Magda wrote:> On Jul 12, 2006, at 20:10, Richard Elling wrote: > >> G1. for 3 <= Ndisks <=5, RAID-Z2 offers the best resiliency/space. For >> Ndisks > 5, RAID-1 offers best resiliency/space. Ndisk-way RAID-1 >> always wins the resiliency, availability, and MTTDL but most people >> won''t do it for Ndisks > 2. > > Can you give some numeric examples for one or two cases? I''m having > troubling ''visualizing'' why this would be the case (especially for N > 5).For a binary resiliency analysis, 8 disks -> 2^8 possible working/failed states. Of those, a RAID-1+0 zpool will survive in 31.6% of the states versus only 14.5% for raidz2. This is a mathematical description of the sometimes more intuitive (sometimes not so intuitive) notion that the raidz2 zpool can survive if any 2 disks fail, whereas the RAID-1+0 zpool can survive up to 4 failed disks, as long as the failed disks are not in a single mirrored pair. Such analysis has nothing to do with availability, it is just a point-in-time assessment of resiliency. -- richard
> > But presumably it would be possible to use additional columns for future > > writes? > > I guess that could be made to work, but then the data on the disk becomes > much (much much) more difficult to interpret because you have some rows which > are effectively one width and others which are another (ad infinitum).How do rows come into it? I was just assuming that each (existing) in-use disk block was pointed to by a FS block, which was tracked by other structures. I was guessing that adding space (effectively extending the "rows") wasn''t going to be noticed for accessing old data. Even after reading through the on-disk format document and trying to comprehend some of the code, I have no idea how the free space is tracked or examined when a raidz block is being allocated on disk.> It also doesn''t really address the issue since you assume that you > want to add space because the disks are getting full, but this scheme, > as you mention, only applies the new width to empty rows.Correct. It would certainly be somewhat of a limitation. But a little bit of block turnover (either through normal usage or explicitly driven) would make it much less of an effect. I''m not suggesting that this is something that should (or could) be implemented. Just that instead of it being technically difficult, it appeared pretty straightforward to me given what little I know about how the disk blocks are currently managed. I''m just trying to understand if this is true or what other bits I''m still misunderstanding. :-) -- Darren Dunham ddunham at taos.com Senior Technical Consultant TAOS http://www.taos.com/ Got some Dr Pepper? San Francisco, CA bay area < This line left intentionally blank to confuse you. >
> > I guess that could be made to work, but then the data on > > the disk becomes much (much much) more difficult to > > interpret because you have some rows which are effectively > > one width and others which are another (ad infinitum). > > How do rows come into it? I was just assuming that each > (existing) in-use disk block was pointed to by a FS block, > which was tracked by other structures. I was guessing that > adding space (effectively extending the "rows") wasn''t > going to be noticed for accessing old data.That''s what my assumption was too. I had the impression from the initial information (I nearly said hype) about ZFS, that the distinctions between RAID levels were to become less clear i.e. that you could have some files stored with higher resilience than others. Maybe this is a dumb question, but I''ve never written a filesystem is there a fundamental reason why you cannot have some files mirrored, with others as raidz, and others with no resilience? This would allow a pool to initially exist on one disk, then gracefully change between different resilience strategies as you add disks and the requirements change. Apologies if this is pie in the sky. Steve.
> Maybe this is a dumb question, but I''ve never written a > filesystem is there a fundamental reason why you cannot have > some files mirrored, with others as raidz, and others with no > resilience? This would allow a pool to initially exist on one > disk, then gracefully change between different resilience > strategies as you add disks and the requirements change.Actually, it''s an excellent question. And a deep one. It goes to the very heart of why the traditional factoring of storage into filesystems and volumes is such a bad idea. In a typical filesystem, each block is represented by a small integer -- typically 32 or 64 bits -- indicating its location on disk. To make a filesystem talk to multiple disks, you either need to add another integer -- a device number -- to each block pointer, or you need to generate virtual block numbers. Doing the former requires modifying the filesystem; doing the latter does not, which is why volumes caught on in the first place. It was expedient. The simplest example of block virtualization is a concatentation of two disks. For simplicity, assume all disks have 100 blocks. To create a 200-block volume using disks A and B, we assign virtual blocks 0-99 to A and 100-199 to B. As far as the filesystem is concerned, it''s just looking at a 200-block logical device. But when it issues a read for (say) logical block 137, the volume manager will actually map that to physical block 37 of disk B. A stripe (RAID-0) is similar, except that instead of putting the low blocks on A and the high ones on B, you put the even ones on A and the odd ones on B. So disk A stores virtual blocks 0, 2, 4, 6, ... on physical blocks 0, 1, 2, 3, etc. The advantage of striping is that when you issue a read of (say) 10 blocks, that maps into 5 blocks on each disk, and you can read from those disks in parallel. So you get up to double the bandwidth (less for small I/O, because then the per-I/O overhead dominates, but I digress). A mirror (RAID-1) is even simpler -- it''s just a 1-1 mapping of logical to physical block numbers on two or more disks. RAID-4 is only slightly more complex. The rule here is that all disks XOR to zero (i.e., if you XOR the nth block of each disk together, you get a block of zeroes), so you can lose any one disk and still be able to reconstruct the data. The block mapping is just like a stripe, except that there''s a parity disk as well. RAID-5 is like RAID-4, but the parity rotates at some fixed interval so that you don''t have a single ''hot'' parity disk. RAID-6 is a variant on RAID-4/5 that (using a bit subtler mathematics) can survive two disk failures, not just one. Now here''s the key limitation of this scheme, which is so obvious that it''s easy to miss: the relationship between replicas of your data is expressed in terms of the *devices*, not the *data*. That''s why a traditional filesystem can''t offer different RAID levels using the same devices -- because the RAID levels are device-wide in nature. In a mirror, all disks are identical. In a RAID-4/5 group, all disks XOR to zero. Mixing (say) mirroring with RAID-5 doesn''t work because in the event of disk failure, the volume manager would have no idea how to reconstruct missing data. RAID-Z takes a different approach. We were designing a filesystem as well, so we could make the block pointers as semantically rich as we wanted. To that end, the block pointers in ZFS contains data layout information. One nice side effect of this is that we don''t need fixed-width RAID stripes. If you have 4+1 RAID-Z, we''ll store 128k as 4x32k plus 32k of parity, just like any RAID system would. But if you only need to store 3 sectors, we won''t do a partial-stripe update of an existing 5-wide stripe; instead, we''ll just allocate four sectors, and store the data and its parity. The stripe width is variable on a per-block basis. And, although we don''t support it yet, so is the replication model. The rule for how to reconstruct a given block is described explicitly in the block pointer, not implicitly by the device configuration. So to answer your question: no, it''s no pie in the sky. It''s a great idea. Per-file or even per-block replication is something we''ve thought about in depth, built into the on-disk format, and plan to support in the future. The main issues are administrative. ZFS is all about ease of use (when it''s not busy being all about data integrity), so getting the interface to be simple and intuitive is important -- and not as simple as it sounds. If your free disk space might be used for single-copy data, or might be used for mirrored data, then how much free space do you have? Questions like that need to be answered, and answered in ways that make sense. (Note: would anyone ever really want per-block replication levels? It''s not as crazy as it sounds. A couple of examples: replicating only the first block, so that even if you lose data, you know the file type and some idea what it contained; replicating only the first (say) 1GB, so that most files are replicated, but giant mpegs and core files aren''t; or in a database, replicating only those records that have a particular field set.) Jeff
If it was possible to implement raidz/raidz2 expansion it would be a big feature in favor of ZFS. Most hardware RAID controllers have the ability to expand a raid pool - some have to take the raid array offline, but the ones I work with generally do it online, although you are forced to suffer through reduced performance until it is done. I''d rather not use ZFS on top of a raid controller. I like the way ZFS works, and putting another layer between it and the disks seems like it would only reduce it''s effectiveness. I''m not saying that expanding a raidz2 would be good for MTTDL, but have that ability there would be very handy.> I''m not suggesting that this is something that should > (or could) be > implemented. Just that instead of it being > technically difficult, it > appeared pretty straightforward to me given what > little I know about how > the disk blocks are currently managed. I''m just > trying to understand if > this is true or what other bits I''m still > misunderstanding. :-) >This message posted from opensolaris.org
David Abrahams wrote:> I''ve seen people wondering if ZFS was a scam because the claims just > seemed too good to be true. Given that ZFS *is* really great, I don''t > think it would hurt to prominently advertise limitations like this one > it would probably benefit credibility considerably, and it''s a real > consideration for anyone who''s doing RAID-Z. > >Very true. I recently had someone imply to me that ZFS was a network protocol and everything else related to disks and file sharing -- instead of volume manager integrated with a filesystem and an automounter. There is hype and misinformation out there. As for the claims, I don''t buy that it''s impossible to corrupt a ZFS volume. I''ve replicated the demo where the guy dd''s /dev/urandom over part of the disk, and I believe that works -- but there are a lot of other ways to corrupt a filesystem in the real world. I''m spending this morning setting up a server to try ZFS in our environment -- which will put it under a heavy load with a lot of large files and heavy churn. We''ll see what happens! -Luke -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3271 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060713/f8eb2b07/attachment.bin>
Luke Scharf <lscharf at vt.edu> writes:> As for the claims, I don''t buy that it''s impossible to corrupt a ZFS > volume. I''ve replicated the demo where the guy dd''s /dev/urandom > over part of the disk, and I believe that works -- but there are a > lot of other ways to corrupt a filesystem in the real world. I''m > spending this morning setting up a server to try ZFS in our > environment -- which will put it under a heavy load with a lot of > large files and heavy churn. We''ll see what happens!I''ve done that one too. It''s fun -- and caused me to learn the difference between /dev/random and /dev/urandom :-). It''s easy to corrupt the volume, though -- just copy random data over *two* disks of a RAIDZ volume. Okay, you have to either do the whole volume, or get a little lucky to hit both copies of some piece of information before you get corruption. Or pull two disks out of the rack at once. With the transactional nature and rotating pool of top-level blocks, I think it will be pretty darned hard to corrupt a structure *short of* deliberate damage exceeding the redundancy of the vdev. If you succeed, you''ve found a bug, don''t forget to report it! -- David Dyer-Bennet, <mailto:dd-b at dd-b.net>, <http://www.dd-b.net/dd-b/> RKBA: <http://www.dd-b.net/carry/> Pics: <http://dd-b.lighthunters.net/> <http://www.dd-b.net/dd-b/SnapshotAlbum/> Dragaera/Steven Brust: <http://dragaera.info/>
Adam Leventhal <ahl at eng.sun.com> writes:> I''m not sure I even agree with the notion that this is a real > problem (and if it is, I don''t think is easily solved). Stripe > widths are a function of the expected failure rate and fault domains > of the system which tend to be static in nature. A coarser solution > would be to create a new pool where you zfs send/zfs recv the > filesystems of the old pool.RAIDZ expansion is a big enough deal that I may end up buying an Infrant NAS box and using their X-RAID instead. The ZFS should be more secure, and I *really* like the block checksumming -- but the ability to expand my existing pool by just adding a new disk is REALLY REALLY USEFUL in a small office or home configuration. Having disks down for hours at work while they arrange to make them bigger suggests there''d be benefits in that market, too. I see phrases like "just add another 7-disk RAIDZ", and I laugh; the boxes I''m looking at mostly have *4* or *5* hot-swap bays. If I could, I''d start with a 2-disk RAIDZ, planning to expand it twice before hitting the system config limit. A *single* 7-disk RAIDZ is probably beyond my means; two of them is absurd to even consider. Possibly this isn''t the market ZFS will make money in, but it''s the market *I''m* in. -- David Dyer-Bennet, <mailto:dd-b at dd-b.net>, <http://www.dd-b.net/dd-b/> RKBA: <http://www.dd-b.net/carry/> Pics: <http://dd-b.lighthunters.net/> <http://www.dd-b.net/dd-b/SnapshotAlbum/> Dragaera/Steven Brust: <http://dragaera.info/>
On Thu, 13 Jul 2006, David Dyer-Bennet wrote:> Adam Leventhal <ahl at eng.sun.com> writes: > > > I''m not sure I even agree with the notion that this is a real > > problem (and if it is, I don''t think is easily solved). Stripe > > widths are a function of the expected failure rate and fault domains > > of the system which tend to be static in nature. A coarser solution > > would be to create a new pool where you zfs send/zfs recv the > > filesystems of the old pool. > > RAIDZ expansion is a big enough deal that I may end up buying an > Infrant NAS box and using their X-RAID instead. The ZFS should be > more secure, and I *really* like the block checksumming -- but the > ability to expand my existing pool by just adding a new disk is REALLY > REALLY USEFUL in a small office or home configuration.The economics of what you''re saying don''t make sense to me. Let me explain: I just did a quick grep for pricing on an Infrant X6 box at $700 with one 250Gb drive installed. And then you would still have to add at least one more disk drive to get data resilience. I picked up two Seagate 500Gb drives (in Frys, on special) for $189 each. So with a $700 budget and ZFS .....> Having disks down for hours at work while they arrange to make them > bigger suggests there''d be benefits in that market, too. > > I see phrases like "just add another 7-disk RAIDZ", and I laugh; the > boxes I''m looking at mostly have *4* or *5* hot-swap bays. If I > could, I''d start with a 2-disk RAIDZ, planning to expand it twice > before hitting the system config limit. A *single* 7-disk RAIDZ is > probably beyond my means; two of them is absurd to even consider. > > Possibly this isn''t the market ZFS will make money in, but it''s the > market *I''m* in. > --Regards, Al Hopper Logical Approach Inc, Plano, TX. al at logical-approach.com Voice: 972.379.2133 Fax: 972.379.2134 Timezone: US CDT OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005 OpenSolaris Governing Board (OGB) Member - Feb 2006
Jeff Bonwick said:> RAID-Z takes a different approach. We were designing a filesystem > as well, so we could make the block pointers as semantically rich > as we wanted. To that end, the block pointers in ZFS contains data > layout information. One nice side effect of this is that we don''t > need fixed-width RAID stripes. If you have 4+1 RAID-Z, we''ll store > 128k as 4x32k plus 32k of parity, just like any RAID system would. > But if you only need to store 3 sectors, we won''t do a partial-stripe > update of an existing 5-wide stripe; instead, we''ll just allocate > four sectors, and store the data and its parity. The stripe width > is variable on a per-block basis. And, although we don''t support it > yet, so is the replication model. The rule for how to reconstruct > a given block is described explicitly in the block pointer, not > implicitly by the device configuration.Thanks for the explanation - a great help in understanding how all this stuff fits together. Unfortunately I''m now less sure about why you cannot ''just'' add another disk to a RAID-Z pool. Is this just a policy decision for the sake of keeping it simple, rather than a technical restriction?> If your free disk space might be used for single-copy data, > or might be used for mirrored data, then how much free space > do you have? Questions like that need to be answered, and > answered in ways that make sense.They need to be answered, but as the storage is scaled up we don''t need any extra accuracy - knowing that a filesystem is somewhere around 80% full is just fine - I really don''t need to care precisely how many blocks are free, and it actually hinders me if I get given the exact information (I have to scale it into the number of GB, or the percentage of space used). The fact that we then pretty much ignore exact block counts then leads me to think that we don''t actually need to care about exactly how many blocks are free on a disk - so if I store N blocks of data it''s acceptable for the number of free blocks to change by domething different to N. And once data starts to be compressed the direct correlation between the size of a file and the amount of disk space it uses goes away in any case. All pretty exciting - how long are we going to have to wait for this? Steve.
David Dyer-Bennet wrote:> It''s easy to corrupt the volume, though -- just copy random data over > *two* disks of a RAIDZ volume. Okay, you have to either do the whole > volume, or get a little lucky to hit both copies of some piece of > information before you get corruption. Or pull two disks out of the > rack at once. >I tried that too - some of the files were borked, but I was impressed that other files on the volume were still recoverable. Also, ZFS automatically started the scrub - which was handy. Unfortunately, my test system only had one HDD (with 3 partitions simulating a RAID-Z), so the timing wasn''t realistic.> With the transactional nature and rotating pool of top-level blocks, I > think it will be pretty darned hard to corrupt a structure *short of* > deliberate damage exceeding the redundancy of the vdev. If you > succeed, you''ve found a bug, don''t forget to report it! >I buy "very good, backed by good theory and good coding". After after a few months of testing, I might even buy "better than any other general purpose filesystem and volume manager". But infallible? If so, I shall name my storage server "Titanic". -Luke -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3271 bytes Desc: S/MIME Cryptographic Signature URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060713/4f98c552/attachment.bin>
On Thu, Jul 13, 2006 at 09:44:18AM -0500, Al Hopper wrote:> On Thu, 13 Jul 2006, David Dyer-Bennet wrote: > > > Adam Leventhal <ahl at eng.sun.com> writes: > > > > > I''m not sure I even agree with the notion that this is a real > > > problem (and if it is, I don''t think is easily solved). Stripe > > > widths are a function of the expected failure rate and fault domains > > > of the system which tend to be static in nature. A coarser solution > > > would be to create a new pool where you zfs send/zfs recv the > > > filesystems of the old pool. > > > > RAIDZ expansion is a big enough deal that I may end up buying an > > Infrant NAS box and using their X-RAID instead. The ZFS should be > > more secure, and I *really* like the block checksumming -- but the > > ability to expand my existing pool by just adding a new disk is REALLY > > REALLY USEFUL in a small office or home configuration. > > The economics of what you''re saying don''t make sense to me. Let me > explain: I just did a quick grep for pricing on an Infrant X6 box at $700 > with one 250Gb drive installed. And then you would still have to add at > least one more disk drive to get data resilience. I picked up two Seagate > 500Gb drives (in Frys, on special) for $189 each. So with a $700 budget > and ZFS .....My figures suggest that an Infrant NV box costs about $650 with no disks. A server box to run Solaris on with hot-swap racks and such costs out to roughly $1600 with no disks, plus a huge learning curve for me. The fact that I''m willing to even *consider* that route says something about how attractive other features of ZFS are to me.... -- David Dyer-Bennet, <mailto:dd-b at dd-b.net>, <http://www.dd-b.net/dd-b/> RKBA: <http://www.dd-b.net/carry/> Pics: <http://dd-b.lighthunters.net/> <http://www.dd-b.net/dd-b/SnapshotAlbum/> Dragaera/Steven Brust: <http://dragaera.info/>
Jeff Bonwick <bonwick at zion.eng.sun.com> writes:> The main issues are administrative. ZFS is all about ease of use > (when it''s not busy being all about data integrity), so getting the > interface to be simple and intuitive is important -- and not as > simple as it sounds. If your free disk space might be used for > single-copy data, or might be used for mirrored data, then > how much free space do you have? Questions like that need > to be answered, and answered in ways that make sense.It seems, on the face of it, as though a *single* sensible answer might be impossible. But it also seems like it might be unnecessary. How often is it useful for a program to ask about the amount of free memory these days? Not often; another process or thread can come along and allocate (or free) memory before the information is used, and make it totally useless. So if you want to deliver one answer, maybe report the maximum amount available under all allowed storage schemes, because it doesn''t matter all that much; you have to be able to fail to allocate it dynamically anyhow. ZFS would need another interface that allows more sophisticated queries. -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Dyer-Bennet <dd-b at dd-b.net> writes:> Adam Leventhal <ahl at eng.sun.com> writes: > >> I''m not sure I even agree with the notion that this is a real >> problem (and if it is, I don''t think is easily solved). Stripe >> widths are a function of the expected failure rate and fault domains >> of the system which tend to be static in nature. A coarser solution >> would be to create a new pool where you zfs send/zfs recv the >> filesystems of the old pool. > > RAIDZ expansion is a big enough deal that I may end up buying an > Infrant NAS box and using their X-RAID instead. The ZFS should be > more secure, and I *really* like the block checksumming -- but the > ability to expand my existing pool by just adding a new disk is REALLY > REALLY USEFUL in a small office or home configuration.Yes, and while it''s not an immediate showstopper for me, I''ll want to know that expansion is coming imminently before I adopt RAID-Z.> I see phrases like "just add another 7-disk RAIDZ", and I laugh; the > boxes I''m looking at mostly have *4* or *5* hot-swap bays. If I > could, I''d start with a 2-disk RAIDZ, planning to expand it twice > before hitting the system config limit. A *single* 7-disk RAIDZ is > probably beyond my means; two of them is absurd to even consider. > > Possibly this isn''t the market ZFS will make money in, but it''s the > market *I''m* in.Ditto, ditto, ditto. -- Dave Abrahams Boost Consulting www.boost-consulting.com
Of course when it''s time to upgrade you can always just call sun and get a Thumper on a "Try before you Buy" - and use it as a temporary storage space for your files while you re-do your raidz/raidz2 virtual device from scratch with an additional disk. zfs send/zfs recieve here I come..... Not sure if Sun would condone this, but how else is a SOHO user going to back up 3TB of data? Here''s my story. I currently have 1.5TB of data scattered throughout my network. Some machines have raid, but a large portion of the data is sitting on standard drives, waiting for them to fail. To help ensure that I don''t lose data, I''m building a NAS. My NAS chassis is a Supermicro SC933T with 15 hot swap SATA spaces. If I use ZFS as it currently exists, I will start off with a 6+2 raidz2 of 500GB drives. I will losing 1TB to parity, but that''s a worthwhile investment. If I add a 5+2 raidz2 of 500GB drives, I''m losing another 1TB to parity, which doesn''t make me particularly happy I''m comfortable with having 2 parity drives for 12 disks, plus another disk for hot spare. Combine that with regular data scrubbing and I wouldn''t be worried about data loss. But without raidz/raidz2 expansion, or a "loaner" storage system, there''s no way to get there short of starting out with all 15 drives. This message posted from opensolaris.org
> Infrant NAS box and using their X-RAID instead.I''ve gone back to solaris from an Infrant box. 1) while the Infrant cpu is sparc, its way, way, slow. a) the web IU takes 3-5 seconds per page b) any local process, rsync, UPnP, SlimServer is cpu starved 2) like a netapp, its frustrating to not have shell access 3) NFSv3 is buggy (use NFSv2) a) http://www.infrant.com/forum/viewtopic.php?t=546 b) NFSv2 works, but its max filesize is 2Gig. 4) 8MB/sec writes and 15MB/sec reads isn''t that fast 5) local rsync writes are 2MB/sec (use NFS instead) put solaris on your our old PPro box. It will be faster (yes!), cheaper and you can do more than one snapshot (and it doesn''t kill the system) plus one gets shell access!
> comfortable with having 2 parity drives for 12 disks,the thread starting config of 4 disks per controller(?): zpool create tank raidz2 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c2t1d0 c2t2d0 then later zpool add tank raidz2 c2t3d0 c2t4d0 c3t1d0 c3t2d0 c3t3d0 c3t4d0 as described, doubles ones IOPs, and usable space in tank, with the loss of another two disks, splitting the cluster into four (and two parity) writes per disk. perhaps a 8 disk controller, and start with zpool create tank raidz c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 then do a zpool add tank raidz c1t6d0 c1t7d0 c1t8d0 c2t1d0 c2t2d0 zpool add tank raidz c2t3d0 c2t4d0 c2t5d0 c2t6d0 c2t7d0 zpool add tank spare c2t8d0 gives one the same largeish cluster size div 4 per raidz disk, 3x the IOPs, less parity math per write, and a hot spare for the same usable space and loss of 4 disks. splitting the max 128k cluster into 12 chunks (+2 parity) makes good MTTR sense but not much performance sense. if someone wants to do the MTTR math between all three configs, I''d love to read it. Rob http://storageadvisors.adaptec.com/2005/11/02/actual-reliability-calculations-for-raid/ http://www.barringer1.com/ar.htm
David Abrahams wrote:> David Dyer-Bennet <dd-b at dd-b.net> writes: > >> Adam Leventhal <ahl at eng.sun.com> writes: >> >>> I''m not sure I even agree with the notion that this is a real >>> problem (and if it is, I don''t think is easily solved). Stripe >>> widths are a function of the expected failure rate and fault domains >>> of the system which tend to be static in nature. A coarser solution >>> would be to create a new pool where you zfs send/zfs recv the >>> filesystems of the old pool. >> RAIDZ expansion is a big enough deal that I may end up buying an >> Infrant NAS box and using their X-RAID instead. The ZFS should be >> more secure, and I *really* like the block checksumming -- but the >> ability to expand my existing pool by just adding a new disk is REALLY >> REALLY USEFUL in a small office or home configuration. > > Yes, and while it''s not an immediate showstopper for me, I''ll want to > know that expansion is coming imminently before I adopt RAID-Z.[in brainstorming mode, sans coffee so far this morning] Better yet, buy two disks, say 500 GByte. Need more space, replace them with 750 GByte, because by then the price of the 750 GByte disks will be as low as the 250 GByte disks today, and the 1.5 TByte disks will be $400. Over time, the cost of disks remains the same, but the density increases. This will continue to occur faster than the development and qualification of complex software. ZFS will already expand a mirror as you replace disks :-) KISS -- richard
On Thu, 2006-07-13 at 11:42 -0700, Richard Elling wrote:> [in brainstorming mode, sans coffee so far this morning] > > Better yet, buy two disks, say 500 GByte. Need more space, replace > them with 750 GByte, because by then the price of the 750 GByte disks > will be as low as the 250 GByte disks today, and the 1.5 TByte disks > will be $400. Over time, the cost of disks remains the same, but the > density increases. This will continue to occur faster than the > development and qualification of complex software. ZFS will already > expand a mirror as you replace disks :-) KISS > -- richardLooking at our (Sun''s) product line now, we''re not just going after the Enterprise market anymore. Specifically, the Medium Business market is a target (few 100 people, a half-dozen IT staff, total). RAIDZ expansion for these folks is essentially a must-have to sell to them. Being able to expand a 2-drive array into a 5-drive RAIDZ by simply pushing in new disks and typing a single command is a HUGE win. Most hardware RAID (even the low-end, and both SCSI & SATA) controllers can do this on-line nowdays. It''s something that is simply expected, and not having it is a big black mark. A typical instance here is a small business server (2-4 CPUs) hooked to a small JBOD. We''re not going to sell them a fully populated JBOD to start with, but selling them one 50% full is much more likely. (look at the price differential between a 3510FC fully and half populated). In the Small Business market, expandability is key, as their limited budgets tend to make for Just-In-Time purchasing. They are _much_ more likely to buy from us things that can be had in a minimum configuration at low cost, but have considerable future expansion, even if the expansion costs them considerably more overall than getting the entire thing in the first place. Also, mixing and matching inside a disk server is unlikely until you get to places that have a highly trained staff. Yes, adding 4 250GB drives is more expensive than adding 2 750GB ones, but it is nominal, compared to the extra effort of configuration and maintenance. At the Medium Business level, less stress on the Admin staff is usually the driving factor after raw cost, since Admin staff tend to be extremely overworked. -- Erik Trimble Java System Support Mailstop: usca14-102 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800)
There''s no reason at all why you can''t do this. The only thing preventing most file systems from taking advantage of ?adjustable? replication is that they don?t have the integrated volume management capabilities that ZFS does. ZFS already allows its storage pools to contain multiple types of block storage (striped, mirrored, RAID, etc.). It?s not much of a stretch to imagine being able to assign, on either a per-filesystem or per-file basis, attributes that would control which of those types a particular file could be stored on. We probably won?t see this soon, but the future is bright! This message posted from opensolaris.org
> Of course when it''s time to upgrade you can always > just call sun and get a Thumper on a "Try before you > Buy" - and use it as a temporary storage space for > your files while you re-do your raidz/raidz2 virtual > device from scratch with an additional disk. zfs > send/zfs recieve here I come.....For all my experience is worth, given enough space, and "just some time to sift through all that and reorganize", data has some mysterical way of growing and populating unused space. Small SOHO sites willing to devote some tender loving care to their accumulated wealth are more prone to that than large sites with established plans and routines. (Though those may compensate for that by having the large device in production for a bit and let their users do the rest - empty disk space has some magnetic attraction to data, it never fails to fill. So that scheme of yours may well end in another Thumper sold, because it''s too populated to migrate and return. ;-) Sorry, I just couldn''t resist. Tatjana This message posted from opensolaris.org
Anton.Rang at Sun.COM said:> There''s no reason at all why you can''t do this. The only thing preventing > most file systems from taking advantage of ?adjustable? replication is that > they don?t have the integrated volume management capabilities that ZFS does.And in fact, Sun''s own QFS can do this, on a per-file (or directory) basis, if one has setup the underlying filesystem with the appropriate mix of replication, striping, etc. Regards, Marion
On Thu, 2006-07-13 at 07:58, David Abrahams wrote:> It seems, on the face of it, as though a *single* sensible answer > might be impossible. But it also seems like it might be unnecessary.I''m aware of at least one case where a customer wrote a "delete file at head of queue; repeat until statvfs says there''s enough free space" loop -- essentially maintaining a FIFO archive of log files -- which was severely confused by one version of logging UFS''s deferred free accounting. Deferred frees caused this loop to kill the entire archive instead of just trimming it. If anyone cares about additional details I can go dig it up -- it was mentioned in a thread on comp.os.solaris a few years back. In busy environments without quotas, the low order bits of a free space count are going to be continuously in flux... but with quotas and fine-grained filesystems, it''s actually IMHO a bit more likely with ZFS than with UFS that you could build something where only one process''s actions could affect the available space reported via statvfs() for a given file system. - Bill
On Thu, Jul 13, 2006 at 11:42:21AM -0700, Richard Elling wrote:> >Yes, and while it''s not an immediate showstopper for me, I''ll want to > >know that expansion is coming imminently before I adopt RAID-Z. > > [in brainstorming mode, sans coffee so far this morning] > > Better yet, buy two disks, say 500 GByte. Need more space, replace > them with 750 GByte, because by then the price of the 750 GByte disks > will be as low as the 250 GByte disks today, and the 1.5 TByte disks > will be $400. Over time, the cost of disks remains the same, but the > density increases. This will continue to occur faster than the > development and qualification of complex software. ZFS will already > expand a mirror as you replace disks :-) KISSindeed, but this is not the same as expansion of an existing vdev because you still have the same number of spindles, with potentially more data on each, so it may in fact be a net performance loss. I don''t think the only driver for wanting to expand a raidz vdev is to gain more space... grant.