Hi there! I am new to the list, and to OpenSolaris, as well as ZPS. I am creating a zpool/zfs to use on my NAS server, and basically I want some redundancy for my files/media. What I am looking to do, is get a bunch of 2TB drives, and mount them mirrored, and in a zpool so that I don''t have to worry about running out of room. (I know, pretty typical I guess). My problem is, is that not all 2TB hard drives are the same size (even though they should be 2 trillion bytes, there is still sometimes a +/- (I''ve only noticed this 2x so far) ) and if I create them mirrored, and one fails, and then I replace the drive, and for some reason, it is 1byte smaller, it will not work. How would I go about fixing this "problem"? ****THIS is just a thought, I am looking for thoughts and opinions on doing this... it prolly would be a bad idea, but hey, does it hurt to ask?**** I have been thinking, and would it be a good idea, to have on the 2TB drives, say 1TB or 500GB "files" and then mount them as mirrored? So basically, have a 2TB hard drive, set up like: (where drive1 and drive2 are the paths to the mount points) Mkfile 465gb /drive1/drive1part1 Mkfile 465gb /drive1/drive1part2 Mkfile 465gb /drive1/drive1part3 Mkfile 465gb /drive1/drive1part4 Mkfile 465gb /drive2/drive2part1 Mkfile 465gb /drive2/drive2part2 Mkfile 465gb /drive2/drive2part3 Mkfile 465gb /drive2/drive2part4 (I use 465gb, as 2TB = 2trillion bytes, / 4 = 465.66 gb) And then add them to the zpool Zpool add medianas mirror /drive1/drive1part1 /drive2/drive2/part1 Zpool add medianas mirror /drive1/drive1part2 /drive2/drive2/part2 Zpool add medianas mirror /drive1/drive1part3 /drive2/drive2/part3 Zpool add medianas mirror /drive1/drive1part4 /drive2/drive2/part4 And then, if a drive goes and I only have a 500gb and a 1.5tb drives, they could be replaced that way? I am sure there are performance issues in doing this, but would the performance outweigh the possibility of hard drive failure and replacing drives? Sorry for posting a novel, but I am just concerned about failure on bigger drives, and putting my media/files into basically what consists of a JBOD type array (on steroids). Steve
On May 2, 2010, at 8:47 AM, Steve Staples wrote:> Hi there! > > I am new to the list, and to OpenSolaris, as well as ZPS. > > I am creating a zpool/zfs to use on my NAS server, and basically I want some > redundancy for my files/media. What I am looking to do, is get a bunch of > 2TB drives, and mount them mirrored, and in a zpool so that I don''t have to > worry about running out of room. (I know, pretty typical I guess). > > My problem is, is that not all 2TB hard drives are the same size (even > though they should be 2 trillion bytes, there is still sometimes a +/- (I''ve > only noticed this 2x so far) ) and if I create them mirrored, and one fails, > and then I replace the drive, and for some reason, it is 1byte smaller, it > will not work. > > How would I go about fixing this "problem"?This problem is already fixed for you in ZFS. For disk sizes in 2TB it may tolerate difference in size up to approximately a little bit less than half a metaslab size which is currently likely to be 16GB, thus it may tolerate difference in size of up to, say, 7.5GB. I think that in most cases difference in sizes is below that figure. You can see it for yourself: bash-4.0# mkfile -n 2000000000000 d0 bash-4.0# zpool create pool `pwd`/d0 bash-4.0# mkfile -n 1992869543936 d1 bash-4.0# zpool attach pool `pwd`/d0 `pwd`/d1 bash-4.0# zpool status pool pool: pool state: ONLINE scrub: resilver completed after 0h0m with 0 errors on Sun May 2 15:25:24 2010 config: NAME STATE READ WRITE CKSUM pool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 /var/tmp/d0 ONLINE 0 0 0 /var/tmp/d1 ONLINE 0 0 0 83.5K resilvered errors: No known data errors bash-4.0# zpool detach pool `pwd`/d1 So you can see that even though difference in size between d0 and d1 is 7130456064 (~6.6GB), it can still be attached just fine. Let''s now detach d1 and make it 1 byte smaller: bash-4.0# mkfile -n 1992869543935 d1 bash-4.0# zpool attach pool `pwd`/d0 `pwd`/d1 cannot attach /var/tmp/d1 to /var/tmp/d0: device is too small bash-4.0# This time is is no longer possible to attach it, because size is not enough to fit the same number (116) of 16G metaslabs;> ****THIS is just a thought, I am looking for thoughts and opinions on doing > this... it prolly would be a bad idea, but hey, does it hurt to ask?**** > > I have been thinking, and would it be a good idea, to have on the 2TB > drives, say 1TB or 500GB "files" and then mount them as mirrored? So > basically, have a 2TB hard drive, set up like: > > (where drive1 and drive2 are the paths to the mount points) > Mkfile 465gb /drive1/drive1part1 > Mkfile 465gb /drive1/drive1part2 > Mkfile 465gb /drive1/drive1part3 > Mkfile 465gb /drive1/drive1part4 > > Mkfile 465gb /drive2/drive2part1 > Mkfile 465gb /drive2/drive2part2 > Mkfile 465gb /drive2/drive2part3 > Mkfile 465gb /drive2/drive2part4 > > (I use 465gb, as 2TB = 2trillion bytes, / 4 = 465.66 gb) > > And then add them to the zpool > Zpool add medianas mirror /drive1/drive1part1 /drive2/drive2/part1 > Zpool add medianas mirror /drive1/drive1part2 /drive2/drive2/part2 > Zpool add medianas mirror /drive1/drive1part3 /drive2/drive2/part3 > Zpool add medianas mirror /drive1/drive1part4 /drive2/drive2/part4This is not a good idea regards victor> And then, if a drive goes and I only have a 500gb and a 1.5tb drives, they > could be replaced that way? > > I am sure there are performance issues in doing this, but would the > performance outweigh the possibility of hard drive failure and replacing > drives? > > Sorry for posting a novel, but I am just concerned about failure on bigger > drives, and putting my media/files into basically what consists of a JBOD > type array (on steroids). > > Steve > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Steve Staples > > My problem is, is that not all 2TB hard drives are the same size (even > though they should be 2 trillion bytes, there is still sometimes a +/- > (I''ve > only noticed this 2x so far) ) and if I create them mirrored, and one > fails, > and then I replace the drive, and for some reason, it is 1byte smaller, > it > will not work.If you use the latest developer release from genunix.org, it''s already fixed. This is what I recommend, in general. http://genunix.org/dist/indiana/ It looks like the latest one is osol-dev-134 If you use the latest opensolaris release (2009.06) I''m not sure if it''s already fixed. If you use Solaris 10, even fully updated, you are wise to ask this question. Because the fix is not present. If you don''t already have the fix, the recommendation would be to partition the disks first. Throw away something like 1% of the disk intentionally. Yes this is extra work and a little wasteful, but it''s preventive work for a real problem. http://docs.sun.com/app/docs/doc/806-4073/6jd67r9hu (Quoting Richard Elling) CR 6844090, zfs should be able to mirror to a smaller disk http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6844090 b117, June 2009
> On May 2, 2010, at 8:47 AM, Steve Staples wrote: > > > Hi there! > > > > I am new to the list, and to OpenSolaris, as well as ZPS. > > > > I am creating a zpool/zfs to use on my NAS server, and basically I wantsome> > redundancy for my files/media. What I am looking to do, is get a bunchof> > 2TB drives, and mount them mirrored, and in a zpool so that I don''t haveto> > worry about running out of room. (I know, pretty typical I guess). > > > > My problem is, is that not all 2TB hard drives are the same size (even > > though they should be 2 trillion bytes, there is still sometimes a +/-(I''ve> > only noticed this 2x so far) ) and if I create them mirrored, and onefails,> > and then I replace the drive, and for some reason, it is 1byte smaller,it> > will not work. > > > > How would I go about fixing this "problem"? > > This problem is already fixed for you in ZFS. For disk sizes in 2TB it may > tolerate difference in size up to approximately a little bit less thanhalf a> metaslab size which is currently likely to be 16GB, thus it may tolerate > difference in size of up to, say, 7.5GB. > > I think that in most cases difference in sizes is below that figure. > > You can see it for yourself: > > bash-4.0# mkfile -n 2000000000000 d0 > bash-4.0# zpool create pool `pwd`/d0 > bash-4.0# mkfile -n 1992869543936 d1 > bash-4.0# zpool attach pool `pwd`/d0 `pwd`/d1 > bash-4.0# zpool status pool > pool: pool > state: ONLINE > scrub: resilver completed after 0h0m with 0 errors on Sun May 2 15:25:24 > 2010 > config: > > NAME STATE READ WRITE CKSUM > pool ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > /var/tmp/d0 ONLINE 0 0 0 > /var/tmp/d1 ONLINE 0 0 0 83.5K resilvered > > errors: No known data errors > bash-4.0# zpool detach pool `pwd`/d1 > > So you can see that even though difference in size between d0 and d1 is > 7130456064 (~6.6GB), it can still be attached just fine. Let''s now detachd1> and make it 1 byte smaller:I''ve done this in testing already, and mirrored a 10gb drive, with an 80gb drive, and yes, the mirror assumes the 10gb size. But, if I remove the 10gb drive, and replace it with another 80gb, if the other 80gb drive is smaller by even 1 byte, you cannot attach it.> bash-4.0# mkfile -n 1992869543935 d1 > bash-4.0# zpool attach pool `pwd`/d0 `pwd`/d1 > cannot attach /var/tmp/d1 to /var/tmp/d0: device is too small > bash-4.0# > > This time is is no longer possible to attach it, because size is notenough to> fit the same number (116) of 16G metaslabs;And yes, I''ve gotten the exact same results when using file based "disks" (mkfile -n 1992869543935 d1) of 1byte less. Replacing with a bigger drive, is no problem, replacing with the same drive, that is 1byte smaller... cannot do.
> > From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > > bounces at opensolaris.org] On Behalf Of Steve Staples > > > > My problem is, is that not all 2TB hard drives are the same size (even > > though they should be 2 trillion bytes, there is still sometimes a +/- > > (I''ve > > only noticed this 2x so far) ) and if I create them mirrored, and one > > fails, > > and then I replace the drive, and for some reason, it is 1byte smaller, > > it > > will not work. > > If you use the latest developer release from genunix.org, it''s already > fixed. This is what I recommend, in general. > http://genunix.org/dist/indiana/ It looks like the latest one is > osol-dev-134 > > If you use the latest opensolaris release (2009.06) I''m not sure if it''s > already fixed.I am currently using OpenSolaris 2009.06 If I was to upgrade to the current "developer" version, forgive my ignorance (since I am new to *solaris), but how would I do this? Also, are there docs for this yet on what has changed? I am looking in the directory there, and I don''t see any "notes". Any gentle nudge in the right direction would be appreciated :)> If you use Solaris 10, even fully updated, you are wise to ask this > question. Because the fix is not present. > > If you don''t already have the fix, the recommendation would be topartition> the disks first. Throw away something like 1% of the disk intentionally. > Yes this is extra work and a little wasteful, but it''s preventive work fora> real problem. http://docs.sun.com/app/docs/doc/806-4073/6jd67r9huIf I was to take this route, I was under the impression that ZFS takes the drive as a whole, regardless of partitions, and wipes the data? Apparently I am wrong? I don''t have any problems with losing a few gig if needed, to save future problems. If I have to partition a drive of 2tb to 1.8tb (2 trillion bytes = 1.862g), then oh well.> (Quoting Richard Elling) > CR 6844090, zfs should be able to mirror to a smaller disk > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6844090 > b117, June 2009Steve
> I am currently using OpenSolaris 2009.06 > If I was to upgrade to the current "developer" version, forgive my > ignorance > (since I am new to *solaris), but how would I do this?# pkg set-publisher -O http://pkg.opensolaris.org/dev opensolaris.org # pkg image-update That''ll take you to snv_134 or whatever the latest is roy
> From: Roy Sigurd Karlsbakk [mailto:roy at karlsbakk.net] > Sent: Sunday, May 02, 2010 11:55 AM > > > I am currently using OpenSolaris 2009.06 > > If I was to upgrade to the current "developer" version, forgive my > > ignorance > > (since I am new to *solaris), but how would I do this? > > # pkg set-publisher -O http://pkg.opensolaris.org/dev opensolaris.org > # pkg image-update > > That''ll take you to snv_134 or whatever the latest isThis may be perfectly correct, but my personal experience is to say that installing from scratch with the later ISO is much more reliable than applying this upgrade path.
> From: Steve Staples [mailto:thestapler.ca at gmail.com] > > I am currently using OpenSolaris 2009.06 > If I was to upgrade to the current "developer" version, forgive my > ignorance > (since I am new to *solaris), but how would I do this?If you go to genunix.org (using the URL in my previous email) you can simply download the 134 iso file there. And install the OS from it. If you ever need to remember this, there''s a ridiculously simple way to figure it out each time. Go to http://opensolaris.org (not to be confused with opensolaris.com) Go to the "download" page, and start reading. Before too long, you''ll see some info about downloading and installing, or upgrading to developer releases. Including a reference to genunix.> > Zpool using partitions instead of whole devices > > If I was to take this route, I was under the impression that ZFS takes > the > drive as a whole, regardless of partitions, and wipes the data?This is true, if you specify the whole device. For example, if you format & partition c0t0d0, and then you "zpool create tank c0t0d0", then you''re going to obliterate your format and partitions. However, if you "zpool create tank c0t0d0p1" or "zpool create tank c0t0d0s1" then it will use the slice (partition) instead of the whole device.
OK, I''ve installed OpenSolaris DEV 134, created 2 files. Mkfile 128m /disk1 Mkfile 127m /disk2 Zpool create stapler /disk1 Zpool attach stapler /disk1 /disk2 Cannot attach /disk2 to /disk1: device is too small (that''s what she said.. lol) But, if I created 128m and 128m - 10bytes, it works. I can attach the smaller drive. And if I create 1000m, I can attach a 999m virtual disk... So, my question is, is what is the ratio on this? how much smaller can drive2 be than drive1? I was trying to find developer notes, and what''s been upgraded. but my searching didn''t turn up anything of interest :( Steve another question, is zpool shrinking available?