I?ve spend a few hours reading through the forums and wiki and honestly my head is spinning. I have been trying to study up on either buying or building a box that would allow me to add drives of varying sizes/speeds/brands (adding more later etc) and still be able to use the full space of drives (minus parity? [not sure if I got the terminology right]) with redundancy. I have found the ?all in one? solution being Drobo however it has many caveats such as a proprietary setup, limited number of drives (I am looking to eventually expand over 8 drives), and a price tag that is borderline criminal.>From what I understand using ZFS one could setup something like RAID 6 (RAID-Z2?) but with the ability to use drives of varying sizes/speeds/brands and able to add additional drives later. Am I about right? If so I will continue studying up on this if not then I guess I need to continue exploring different options. Thanks!!Cheers, -Gaiko -- This message posted from opensolaris.org
If you use drives of varying size, zfs will use the smallest capacity drives. Say you have 1TB + 2TB + 2TB, then ZFS create a raid with 1TB large drives. 3 x 1TB raid will be result. One ZFS raid consists of vdevs, that is, a group of drives. That vdev can be configured as raidz1 (raid-5) or raidz2 (raid-6) or if you have two disks: as a mirror. Say you have 8 drives, and you create a zfs raid configured as raidz2 (raid-6). Then you can add a new group of drives, say 3 new drives configured as a raidz1. Then your zfs raid will consists of two groups of drives, 8 drives, and 3 drives. You can never change the number of disks in a group (can not decrease or increase). However, you can add a new group whenever you want. I suggest you install Solaris 11 Express in VirtualBox and play around. You can create empty files in Solaris, and then use them instead of hard drives. So, you can create a zfs raid with 8 files, and play around and make snapshots etc . Or install OpenIndiana, which is the free open sourced version of Solaris 11 Express. -- This message posted from opensolaris.org
Thank you kebabber. I will try out indiana and virtual box to play around with it a bit. Just to make sure I understand your example, if I say had a 4x2tb drives, 2x750gb, 2x1.5tb drives etc then i could make 3 groups (perhaps 1 raidz1 + 1 mirrored + 1 mirrored), in terms of accessing them would they just be mounted like 3 partitions or could it all be accessed like one big partition? Anywho, I have indiana DL''ing now (very slow connection so thought I would post while i wait). Cheers, -Gaiko -- This message posted from opensolaris.org
Yes, you create three groups as you described and insert them into your zpool (the zfs raid). So you have only one ZFS raid, consisting of three groups. You dont have three different ZFS raids (unless you configure that). You can also later, swap one disk to a larger and repair the group. Then you swap the next disk to a larger, etc. When all disks are swapped, the group will be bigger. And remember, you can never change the number of disks in a group. But you can add a new group. And you can also grow the group by swapping each disk to a larger. -- This message posted from opensolaris.org
In zfs terminology each of the groups you have is a VDEV and a zpool can be made of a number of VDEVs. This zpool can then be mounted as a single filesystem, or you can split it into as many filesystems as you wish. So the answer is yes to all the configurations you asked about and a lot more :) Bye, Deano deano at cloudpixies.com -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Gaikokujin Kyofusho Sent: 05 February 2011 17:55 To: zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] ZFS/Drobo (Newbie) Question Thank you kebabber. I will try out indiana and virtual box to play around with it a bit. Just to make sure I understand your example, if I say had a 4x2tb drives, 2x750gb, 2x1.5tb drives etc then i could make 3 groups (perhaps 1 raidz1 + 1 mirrored + 1 mirrored), in terms of accessing them would they just be mounted like 3 partitions or could it all be accessed like one big partition? Anywho, I have indiana DL''ing now (very slow connection so thought I would post while i wait). Cheers, -Gaiko -- This message posted from opensolaris.org _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Sat, Feb 5, 2011 at 9:54 AM, Gaikokujin Kyofusho <gaikokujinkyofusho at gmail.com> wrote:> Just to make sure I understand your example, if I say had a 4x2tb drives, 2x750gb, 2x1.5tb drives etc then i could make 3 groups (perhaps 1 raidz1 + 1 mirrored + 1 mirrored), in terms of accessing them would they just be mounted like 3 partitions or could it all be accessed like one big partition?You could add them to one pool, and then create multiple filesystems inside the pool. You total storage would be the sum of the drives'' capacity after redundancy, or 3x2tb + 750gb + 1.5tb. It''s not recommended to use different levels of redundancy in a pool, so you may want to consider using mirrors for everything. This also makes it easier to add or upgrade capacity later. -B -- Brandon High : bhigh at freaks.com
On Sat, February 5, 2011 03:54, Gaikokujin Kyofusho wrote:> From what I understand using ZFS one could setup something like RAID 6 > (RAID-Z2?) but with the ability to use drives of varying > sizes/speeds/brands and able to add additional drives later. Am I about > right? If so I will continue studying up on this if not then I guess I > need to continue exploring different options. Thanks!!IMHO, your best bet for this kind of configuration is to use mirror pairs, not RAIDZ*. Because... Things you can''t do with RAIDZ*: You cannot remove a vdev from a pool. You cannot make a RAIDZ* vdev smaller (fewer disks). You cannot make a RAIDZ* vdev larger (more disks). To increase the storage capacity of a RAIDZ* vdev you need to replace all the drives, one at a time, waiting for resilver between replacements (resilver times can be VERY long with big modern drives). And during each resilver, your redundancy will be reduced by 1 -- meaning a RAIDZ array would have NO redundancy during the resilver. (And activity in the pool is high during the resilver -- meaning the chances of any marginal drive crapping out are higher than normal during the resilver.) With mirrors, you can add new space by adding simply two drives (add a new mirror vdev). You can upgrade an existing mirror by replacing only two drives. You can upgrade an existing mirror without reducing redundancy below your starting point ever -- you attach a new drive, wait for the resilver to complete (at this point you have a three-way mirror), then detach one of the original drives; repeat for another new drive and the other original drive. Obviously, using mirrors requires you to buy more drives for any given amount of usable space. I must admit that my 8-bay hot-swap ZFS server cost me a LOT more than a Drobo (but then I bought in 2006, too). -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
On Sat, February 5, 2011 11:54, Gaikokujin Kyofusho wrote:> Thank you kebabber. I will try out indiana and virtual box to play around > with it a bit. > > Just to make sure I understand your example, if I say had a 4x2tb drives, > 2x750gb, 2x1.5tb drives etc then i could make 3 groups (perhaps 1 raidz1 + > 1 mirrored + 1 mirrored), in terms of accessing them would they just be > mounted like 3 partitions or could it all be accessed like one big > partition?A ZFS pool can contain many vdevs; you could put the three groups you describe into one pool, and then assign one (or more) file-systems to that pool. Putting them all in one pool seems to me the natural way to handle it; they''re all similar levels of redundancy. It''s more flexible to have everything in one pool, generally. (You could also make separate pools; my experience, for what it''s worth, argues for making pools based on redundancy and performance (and only worry about BIG differences), and assign file-systems to pools based on needs for redundancy and performance. And for my home system I just have one big data pool, currently consisting of 1x1TB, 2x400GB, 2x400GB, plus 1TB hot spare.) Or you could stick strictly to mirrors; 4 pools 2x2T, 2x2T, 2x750G, 2x1.5T. Mirrors are more flexible, give you more redundancy, and are much easier to work with. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
On Mon, February 7, 2011 14:59, David Dyer-Bennet wrote:> > On Sat, February 5, 2011 11:54, Gaikokujin Kyofusho wrote: >> Thank you kebabber. I will try out indiana and virtual box to play >> around >> with it a bit. >> >> Just to make sure I understand your example, if I say had a 4x2tb >> drives, >> 2x750gb, 2x1.5tb drives etc then i could make 3 groups (perhaps 1 raidz1 >> + >> 1 mirrored + 1 mirrored), in terms of accessing them would they just be >> mounted like 3 partitions or could it all be accessed like one big >> partition? > > A ZFS pool can contain many vdevs; you could put the three groups you > describe into one pool, and then assign one (or more) file-systems to that > pool. Putting them all in one pool seems to me the natural way to handle > it; they''re all similar levels of redundancy. It''s more flexible to have > everything in one pool, generally. > > (You could also make separate pools; my experience, for what it''s worth, > argues for making pools based on redundancy and performance (and only > worry about BIG differences), and assign file-systems to pools based on > needs for redundancy and performance. And for my home system I just have > one big data pool, currently consisting of 1x1TB, 2x400GB, 2x400GB, plus > 1TB hot spare.)Typo; I don''t in fact have a non-redundant vdev in my main data pool! It''s *2*x1TB at the start of that list. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
> Or you could stick strictly to mirrors; 4 pools 2x2T, 2x2T, 2x750G, > 2x1.5T. Mirrors are more flexible, give you more redundancy, and are > much easier to work with.Easier to work with, yes, but a RAIDz2 will statistically be safer than a set of mirrors, since in many cases, you loose a drive and during resilver, you find bad sectors on another drive in the same VDEV, resulting in data corruption. With RAIDz2 (or 3), the chance of these errors to be on the same place on all drives is quite minimal. With a (striped?) mirror, a single bitflip on the ''healthy'' drive will involve data corruption. Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 roy at karlsbakk.net http://blogg.karlsbakk.net/ -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et element?rt imperativ for alle pedagoger ? unng? eksessiv anvendelse av idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate og relevante synonymer p? norsk.
On Tue, February 8, 2011 13:03, Roy Sigurd Karlsbakk wrote:>> Or you could stick strictly to mirrors; 4 pools 2x2T, 2x2T, 2x750G, >> 2x1.5T. Mirrors are more flexible, give you more redundancy, and are >> much easier to work with. > > Easier to work with, yes, but a RAIDz2 will statistically be safer than a > set of mirrors, since in many cases, you loose a drive and during > resilver, you find bad sectors on another drive in the same VDEV, > resulting in data corruption. With RAIDz2 (or 3), the chance of these > errors to be on the same place on all drives is quite minimal. With a > (striped?) mirror, a single bitflip on the ''healthy'' drive will involve > data corruption.Wait, are you saying that the handling of errors in RAIDZ and mirrors is completely different? That it dumps the mirror disk immediately, but keeps trying to get what it can from the RAIDZ disk? Because otherwise, you assertion doesn''t seem to hold up. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
On Tue, Feb 8, 2011 at 12:53 PM, David Dyer-Bennet <dd-b at dd-b.net> wrote:> Wait, are you saying that the handling of errors in RAIDZ and mirrors is > completely different? ?That it dumps the mirror disk immediately, but > keeps trying to get what it can from the RAIDZ disk? ?Because otherwise, > you assertion doesn''t seem to hold up.I think he meant that if one drive in a mirror dies completely, then any single read error on the remaining drive is not recoverable. With raidz2 (or a 3-way mirror for that matter), if one drive dies completely, you still have redundancy. -B -- Brandon High : bhigh at freaks.com
On 2011-02-08 21:39, Brandon High wrote:> On Tue, Feb 8, 2011 at 12:53 PM, David Dyer-Bennet<dd-b at dd-b.net> wrote: >> Wait, are you saying that the handling of errors in RAIDZ and mirrors is >> completely different? That it dumps the mirror disk immediately, but >> keeps trying to get what it can from the RAIDZ disk? Because otherwise, >> you assertion doesn''t seem to hold up. > > I think he meant that if one drive in a mirror dies completely, then > any single read error on the remaining drive is not recoverable. > > With raidz2 (or a 3-way mirror for that matter), if one drive dies > completely, you still have redundancy.Sure, a 2-way mirror has only 100% redundancy; if one dies, no more redundancy. Same for a RAIDZ -- if one dies, no more redundancy. But a 4-drive RAIDZ has roughly twice the odds of a 2-drive mirror of having a drive die. And sure, a RAIDZ two has more redundancy -- as does a 3-way mirror. Or a 48-way mirror (I read a report from somebody who mirrored all the drives in a Thumper box, just to see if he could). -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info