Guys i have 2 questions. 1. In zfs can you currently add more disks to an existing raidz? This is important to me as i slowly add disks to my system one at a time. 2. in a raidz do all the disks have to be the same size? This is the one thing that has always been a pain with a raid 5 as you need to split the disks into multiple raid 5''s to use all the space. I never understood why the system to not do the same thing behind the scenes for you. EG 320gb x 4 and 1tb x 4. The ultimate is to only lose 1TB to parity instead of 1.32TB. To me it is not the space loss so much as the extra drive slot that i lose. instead i normally just create a raid5 of 320 x 8 and a rait 5 of 680 x 4. I would prefer just to create one raidz with all the disks and let the system work out the best way to get the most data out of it. I did do a test of the raidz with different size disks in a virtual machine and it would not let me create the raidz without using -F. once i created it with -F the pool size seems a little strange and i can''t work out the combo of disks and space lost to parity to get the filesize it returned. Chris This message posted from opensolaris.org
Chris Gilligan <opensolaris at gilligan.id.au> wrote:> 2. in a raidz do all the disks have to be the same size?Related question: Does a raidz have to be either only full disks or only slices, or can it be mixed? E.g., can you do a 3-way raidz with 2 complete disks and one slice (of equal size as the disks) on a 3rd, larger, disk? - Marcus
>> 2. in a raidz do all the disks have to be the same size?Disks don''t have to be the same size, but only as much space will be used on the larger disks will be used as is available on the smallest disk. In other words, there''s no benefit to be gained from this approach.> Related question: > Does a raidz have to be either only full disks or only slices, or can > it be mixed? E.g., can you do a 3-way raidz with 2 complete disks and > one slice (of equal size as the disks) on a 3rd, larger, disk?Sure. One could do this, but it''s kind of a hack. I imagine you''d like to do something like match a disk of size N with another disk of size 2N and use RAID-Z to turn them into a single vdev. At that point it''s probably a better idea to build a striped vdev and use ditto blocks to do your data redundancy by setting copies=2. Adam -- Adam Leventhal, Fishworks http://blogs.sun.com/ahl
ok maybe i should rewrite my question in a better way. My data is mostly made up of things i can afford to lose but would very much not like to lose if a disk dies if at all possible. Due to this i have used a raid5 array in the past. The issue i have had with this is a need to replace all 10 disks at once to increase my storage in the raid and borrow another raid card so i can connect all 20 disks at once while i move the data. What i would like to be able to do is slowly grow my capacity by replacing 320gb disks with 1tb disks one at a time and sometimes adding in extra disks to the system as i can support up to 12 disks. Does anyone have any ideas on the best way to do this with minimum space loss? This just home based storage so i am trying to do everything on the cheap. I thought raidz may have been the answer but that does not seem to be the case. CHris This message posted from opensolaris.org
Chris,
You can replace the disks one at a time with larger disks. No problem.
You can also add another raidz vdev, but you can''t add disks to an
existing raidz vdev.
See the sample output below. This might not solve all your problems,
but should give you some ideas...
Cindy
# zpool create rpool raidz disk01-320 disk02-320 disk03-320 disk04-320 \
disk05-320 disk06-320
# zpool status rpool
pool: rpool
state: ONLINE
scrub: resilver completed with 0 errors on Wed Mar 5 14:36:53 2008
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz1 ONLINE 0 0 0
disk01-320 ONLINE 0 0 0
disk02-320 ONLINE 0 0 0
disk03-320 ONLINE 0 0 0
disk04-320 ONLINE 0 0 0
disk05-320 ONLINE 0 0 0
disk06-320 ONLINE 0 0 0
# zpool replace rpool disk01-320 disk01-1TB
and so on until each disk0*-320 is replaced with disk0*-1TB
# zpool add rpool raidz disk07-1TB disk08-1TB disk09-1TB disk10-1TB \
disk11-1TB disk12-1TB
# zpool status rpool
pool: rpool
state: ONLINE
scrub: resilver completed with 0 errors on Wed Mar 5 14:38:53 2008
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz1 ONLINE 0 0 0
disk01-1TB ONLINE 0 0 0
disk02-1TB ONLINE 0 0 0
disk03-1TB ONLINE 0 0 0
disk04-1TB ONLINE 0 0 0
disk05-1TB ONLINE 0 0 0
disk06-1TB ONLINE 0 0 0
raidz1 ONLINE 0 0 0
disk07-1TB ONLINE 0 0 0
disk08-1TB ONLINE 0 0 0
disk09-1TB ONLINE 0 0 0
disk10-1TB ONLINE 0 0 0
disk11-1TB ONLINE 0 0 0
disk12-1TB ONLINE 0 0 0
Chris Gilligan wrote:> ok maybe i should rewrite my question in a better way.
>
> My data is mostly made up of things i can afford to lose but would very
much not like to lose if a disk dies if at all possible. Due to this i have
used a raid5 array in the past. The issue i have had with this is a need to
replace all 10 disks at once to increase my storage in the raid and borrow
another raid card so i can connect all 20 disks at once while i move the data.
>
> What i would like to be able to do is slowly grow my capacity by replacing
320gb disks with 1tb disks one at a time and sometimes adding in extra disks to
the system as i can support up to 12 disks. Does anyone have any ideas on the
best way to do this with minimum space loss?
>
> This just home based storage so i am trying to do everything on the cheap.
I thought raidz may have been the answer but that does not seem to be the case.
>
> CHris
>
>
> This message posted from opensolaris.org
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> Chris, > > You can replace the disks one at a time with larger > disks. No problem. > You can also add another raidz vdev, but you can''t > add disks to an > existing raidz vdev. > > See the sample output below. This might not solve all > your problems, > but should give you some ideas... > > CindyCindy, What you said would be perfect if i could just replace 1-2 disks at a time (think per month). What i might need to do is split the disks into multi partitions so i can use all the space but then again that wont work as you can not add to a raidz. Also are you sure the size of a raidz will increase if you replace all the disks? If the raidz supports disks of different sizes in the raid i would be set. I noticed some discussion on this topic but for now it seems like it is not looking good. I guess there just is not really any file systems designed for buget file storage expansion. I was really hoping there was a way. but even replacing 5-6 disks at once is too costly for me Chris This message posted from opensolaris.org
Chris Gilligan wrote:> ok maybe i should rewrite my question in a better way. >No, the reason nobody answered was that this a frequent FAQ, second only to CR 4852783 reduce pool capacity. Cindy, can we update the opensolaris.org FAQ to include some words about these two questions? -- richard> My data is mostly made up of things i can afford to lose but would very much not like to lose if a disk dies if at all possible. Due to this i have used a raid5 array in the past. The issue i have had with this is a need to replace all 10 disks at once to increase my storage in the raid and borrow another raid card so i can connect all 20 disks at once while i move the data. > > What i would like to be able to do is slowly grow my capacity by replacing 320gb disks with 1tb disks one at a time and sometimes adding in extra disks to the system as i can support up to 12 disks. Does anyone have any ideas on the best way to do this with minimum space loss? > > This just home based storage so i am trying to do everything on the cheap. I thought raidz may have been the answer but that does not seem to be the case. > > CHris > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
> Chris Gilligan wrote: > > ok maybe i should rewrite my question in a better > way. > > > > No, the reason nobody answered was that this a > frequent FAQ, > second only to CR 4852783 reduce pool capacity. >Famous last words but i thought i read everything in the FAQ but maybe i missed it and i want to make sure i have this right. a raidz is basically constant. you can''t add extra disks, you can''t reduce the number of disks, you can''t increase it size by adding some larger disks and it does not take advantage of say 2x 250gb and 3x 1tb disks. it would treat them all as 250gb. Also from what i have read in other posts there are no plans to change this. Lastly am i also right in saying there is no easy way to replace one raidz of say 6 250gb''s with a new raidz of 4x 1tb disks? liek you replace disks in a raidz but instead replace raidz in a pool thanks Chris This message posted from opensolaris.org
Chris Gilligan wrote:>> Chris Gilligan wrote: >> >>> ok maybe i should rewrite my question in a better >>> >> way. >> >>> >>> >> No, the reason nobody answered was that this a >> frequent FAQ, >> second only to CR 4852783 reduce pool capacity. >> >> > > Famous last words but i thought i read everything in the FAQ but maybe i missed it and i want to make sure i have this right. > > a raidz is basically constant. you can''t add extra disks, you can''t reduce the number of disks, you can''t increase it size by adding some larger disks and it does not take advantage of say 2x 250gb and 3x 1tb disks. it would treat them all as 250gb. > > Also from what i have read in other posts there are no plans to change this. > > Lastly am i also right in saying there is no easy way to replace one raidz of say 6 250gb''s with a new raidz of 4x 1tb disks? liek you replace disks in a raidz but instead replace raidz in a pool >I guess it depends on your definition of "easy." You could back everything up to floppy and restore on a new pool. IMHO it is much easier to just use mirrors which aren''t subject to the set restrictions of raidz or raidz2. -- richard> thanks > > Chris > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Chris, You would need to replace all the disks to see the expanded space. Otherwise, space on the 1-2 larger disks would be wasted. If you replace all the disks with larger disks, then yes, the disk space in the raidz config would be expanded. A ZFS mirrored config would be more flexible but it uses more space, obviously. For example, you could start with a two-disk mirrored config, replace those disks with large disks, eventually add another two-disk mirror, or even add disks to create two 3-way mirrors. Then, change your mind and detach one disk from each mirror. I hope you''ve seen the best practices site. You might get more ideas about whether you want to use slices or not. Eventually, you could replace a sliced config with a whole disk config... http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide Cindy Chris Gilligan wrote:>>Chris, >> >>You can replace the disks one at a time with larger >>disks. No problem. >>You can also add another raidz vdev, but you can''t >>add disks to an >>existing raidz vdev. >> >>See the sample output below. This might not solve all >>your problems, >>but should give you some ideas... >> >>Cindy > > > Cindy, > What you said would be perfect if i could just replace 1-2 disks at a time (think per month). What i might need to do is split the disks into multi partitions so i can use all the space but then again that wont work as you can not add to a raidz. Also are you sure the size of a raidz will increase if you replace all the disks? > > If the raidz supports disks of different sizes in the raid i would be set. I noticed some discussion on this topic but for now it seems like it is not looking good. > > I guess there just is not really any file systems designed for buget file storage expansion. I was really hoping there was a way. but even replacing 5-6 disks at once is too costly for me > > Chris > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> 1. In zfs can you currently add more disks to an existing raidz? This is important to me as i slowly add disks to my system one at a time.No, but solaris and linux raid5 can do this (in linux, grow with mdadm).> 2. in a raidz do all the disks have to be the same size?I think this one has been answered, but I''ll add/ask this: I''m not sure what would happen if you had 3x 320gb and 3x 1tb in a 6 disk raidz array. I know you''d have a 6 * 320gb array, but I don''t know if the unused space on the 3x 1tb could be made into another raidz array. If zfs is limited in this way, you could work around it by making 320gb and 1tb-320gb partitions on the 1tb disks. FYI, if you have some data that doesn''t really need to be redundant, you can simplify your setup with block copies and maybe get closer to raidz efficiency than a straight mirror. Then you could easily replace any disk any time with a bigger one, or add a disk any time. To do this, just make one file system with copies=2 and store important stuff there. Store less important stuff in a copies=1 file system. This message posted from opensolaris.org
> > 2. in a raidz do all the disks have to be the same size? > > > I think this one has been answered, but I''ll add/ask this: I''m not sure what would > happen if you had 3x 320gb and 3x 1tb in a 6 disk raidz array. I know you''d have a > 6 * 320gb array, but I don''t know if the unused space on the 3x 1tb could be made > into another raidz array. If zfs is limited in this way, you could work around it by making > 320gb and 1tb-320gb partitions on the 1tb disks.You have to use partitions.> > FYI, if you have some data that doesn''t really need to be redundant, you can > simplify your setup with block copies and maybe get closer to raidz efficiency than > a straight mirror. Then you could easily replace any disk any time with a bigger one, > or add a disk any time. To do this, just make one file system with copies=2 and > store important stuff there. Store less important stuff in a copies=1 file system.That is bad advice. Both copies may end up on the same disk. zfs will try to put your copies on diffrent disks, but it won''t tell you if it can''t. Use mirror or RAIDZ if your data is important!
On Thu, Mar 6, 2008 at 8:56 PM, MC <rac at eastlink.ca> wrote:> > 1. In zfs can you currently add more disks to an existing raidz? This is important to me > as i slowly add disks to my system one at a time. > > No, but solaris and linux raid5 can do this (in linux, grow with mdadm).Be aware that growing an SLVM / DiskSuite RAID5 doesn''t really grow the RAID5 set, it just concats more components onto the end of it. If those components are mirrors then you still have redundancy, if they aren''t then you don''t for that data that ends up out there. I don''t consider growing RIAD5 this was with DiskSuite a good way to go, short or long term. -- {--------1---------2---------3---------4---------5---------6---------7---------} Paul Kraus -> Sound Designer, Noel Coward''s Hay Fever @ Albany Civic Theatre, Feb./Mar. 2008 -> Facilities Coordinator, Albacon 2008
2008/3/7, Paul Kraus <pk1048 at gmail.com>:> On Thu, Mar 6, 2008 at 8:56 PM, MC <rac at eastlink.ca> wrote: > > > 1. In zfs can you currently add more disks to an existing raidz? This is important to me > > as i slowly add disks to my system one at a time. > > > > No, but solaris and linux raid5 can do this (in linux, grow with mdadm). > > > Be aware that growing an SLVM / DiskSuite RAID5 doesn''t really > grow the RAID5 set, it just concats more components onto the end of > it. If those components are mirrors then you still have redundancy, if > they aren''t then you don''t for that data that ends up out there. I > don''t consider growing RIAD5 this was with DiskSuite a good way to go, > short or long term. >No, that is not how it works. If you grow a RAID5 set the new data is concatenated to the original RAID5, but it IS protected by the parity in the RAID5 set. You have redundancy but not the best performance.>From the docs: http://docs.sun.com/app/docs/doc/816-4520/about-raid5-1?a=viewYou can expand a RAID-5 volume by concatenating additional components to the volume. Concatenating a new component to an existing RAID-5 volume decreases the overall performance of the volume because the data on concatenations is sequential. Data is not striped across all components. The original components of the volume have data and parity striped across all components. This striping is lost for the concatenated component. However, the data is still recoverable from errors because the parity is used during the component I/O. The resulting RAID-5 volume continues to handle a single component failure. Concatenated components also differ in the sense that they do not have parity striped on any of the regions. Thus, the entire contents of the component are available for data. Any performance enhancements for large or sequential writes are lost when components are concatenated.
> > 1. In zfs can you currently add more disks to an existing raidz? > > This is important to me as i slowly add disks to my system one at a > > time. > > NoIs this being worked on? Is it even planned? (I''ve looked at a bunch of FAQs and searched some mailing lists but I can''t find the answers so I ask here.) - Marcus