Folks, All the talk about JBOD and redundancy has me thinking about something. Typically, when I set up a larger system with multiple JBOD devices attached to it, I''d like to set up my mirrors (or RAID-5) to span across the JBODs, to increase my fault-tolerance. Let''s take this use case instance: I have 2 3510FC arrays, no HW raid. Now, I''ve got 12 drives in each array, which, if I hook them up to a machine (using just one FC connection per array, for now), appear to be drives in the form of c2* and c3* (e.g. c2t0d0s2, c2t1d0s2, ... c3t11d0s2 ) Now, in order to mirror this efficiently in a single pool, I need to do: zpool create tank mirror c2t0d0s2 c3t0d0s2 mirror c2t1d0s2 c3t1d0s2 [...] mirror c2t11d0s2 c3t11d0s2 This is a major PITA. Yes, I know this could possibly be scripted. However, it would be REALLY NICE if I could simply do this following: zpool create tank mirror c2 c3 And have ZFS figure out that I want to mirror all drives on c2 to their corresponding part on c3. This does make a couple of assumptions on ZFS''s part (like that there are identical size/number of drives in the corresponding places in both JBODs), but this is not an unreasonable assumption to make. Additionally, after that, perhaps it would be nice to say something like: zpool create tank mirror c2 c3 spares c2t0d0s2 c3t0d0s2 and have it use the first disk in each JBOD as a hot spare, rather than include it in the array. For RAID-Z, it would be really nice to do the corresponding thing (for 3 or more JBODs): zpool create tank raidz c2 c3 c4 (creates a N-way stripe across RAIDZ units which contact 3 members) And, of course, zpool create tank raidz raidz c2 c3 c4 (creates a raidz of raidz units (each subunit with 3 members) and zpool create tank raidz mirror c2 c3 (N-wide raidz of 2-way mirrors) Of course, we''d have to think through all the usage patterns to make sure we get it right (the syntax, I mean), but it would be a big win if we could avoid having to fully specify all devices for some common-use cases. ---------------- On a semi-related note, assuming I have the following: zpool create tank mirror c1t0d0s2 c2t0d0s2 mirror c1t1d0s2 c2t2d0s2 (2-wide stripe of 2-way mirror) I''d like to add a 3rd drive to both mirrors (e.g. if I was creating it from scratch: zpool create tank mirror c1t0d0s2 c2t0d0s2 c3t0d0s2 mirror c1t1d0s2 c2t2d0s2 c3t2d0s2 ) Is this possible to do with an existing pool? This isn''t the most common case (rather, adding to RAIDZ subdevices is the more likely case), but is illustrative of the problem. -Erik -- Erik Trimble Java System Support Mailstop: usca14-102 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800)
Erik Trimble wrote:> zpool create tank mirror c2 c3You can already drop the s? part from the end and just specify the disk but I know thats now that you are looking for. You could: # cd /dev/dsk # zpool create tank mirror c2*s2 c3*s2 :-) But yeah I agree it would be nice to be able to do what you are asking for, particularly with JBODs that present 12 disks (or more) from a single controller. -- Darren J Moffat
On Tue, 2006-04-04 at 09:55, Darren J Moffat wrote:> Erik Trimble wrote: > > > zpool create tank mirror c2 c3 > > You could: > > # cd /dev/dsk > # zpool create tank mirror c2*s2 c3*s2Doesn''t that just create a single 24-way (or whatever) mirror? -- -Peter Tribble L.I.S., University of Hertfordshire - http://www.herts.ac.uk/ http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/
Peter Tribble wrote:> On Tue, 2006-04-04 at 09:55, Darren J Moffat wrote: >> Erik Trimble wrote: >> >>> zpool create tank mirror c2 c3 >> You could: >> >> # cd /dev/dsk >> # zpool create tank mirror c2*s2 c3*s2 > > Doesn''t that just create a single 24-way (or whatever) mirror?Yeah, typo but you get what I meant it is a very clunky workaround! -- Darren J Moffat
On Mon, 2006-04-03 at 22:58, Erik Trimble wrote: ...> Now, in order to mirror this efficiently in a single pool, I need to do: > > zpool create tank mirror c2t0d0s2 c3t0d0s2 mirror c2t1d0s2 c3t1d0s2 > [...] mirror c2t11d0s2 c3t11d0s2 > > > This is a major PITA. Yes, I know this could possibly be scripted. > However, it would be REALLY NICE if I could simply do this following: > > zpool create tank mirror c2 c3 > > > And have ZFS figure out that I want to mirror all drives on c2 to their > corresponding part on c3..Or do you want: zfsassist create -n tank -p 2 -r 2 c2*s2 c3*s2 along the lines of metassist? This isn''t just cosmetically different. It allows you to specify policy - in this case, I force the configuration to be a mirror with 2 copies, and each mirror has to have 2 separate data paths. So this would go away and create a pool just as you described. And would do so without the risk of typos and dumb errors, and with some intelligence as to the sort of best practices that should be adopted. -- -Peter Tribble L.I.S., University of Hertfordshire - http://www.herts.ac.uk/ http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/
On Tue, Apr 04, 2006 at 01:22:24PM +0100, Peter Tribble wrote:> > Or do you want: > > zfsassist create -n tank -p 2 -r 2 c2*s2 c3*s2 > > along the lines of metassist?Ahhhh! Just say no to metassist! If our interface isn''t straightforward enough then something is wrong. See the following RFEs: 6341123 zpool create should allow using wildcards 4868036 zpool create should have better support for total world domination The controller-based syntax has also been proposed before. - Eric -- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
Thanks, Eric, I figured that we''d be revisiting the controller-syntax sooner or later. And I do hate metassist - while the metainit syntax is a little freaky, it''s not sooooo bad once you actually use it for awhile, and as I''d like to see zpool be the sole interface for ZFS admin, keeping it clean is important. Onto my second inquiry (at the bottom of the previous note): can you add drives to a sub-device of a pool? That is, if a ZFS pool is made up of udevs which are RAIDZ or mirrors, can one add disks to those udevs without having to recreate the whole pool? I can see this as being very necessary, when one starts to add whole new JBODs into a config... -Erik On Tue, 2006-04-04 at 09:50 -0700, Eric Schrock wrote:> On Tue, Apr 04, 2006 at 01:22:24PM +0100, Peter Tribble wrote: > > > > Or do you want: > > > > zfsassist create -n tank -p 2 -r 2 c2*s2 c3*s2 > > > > along the lines of metassist? > > Ahhhh! Just say no to metassist! If our interface isn''t > straightforward enough then something is wrong. See the following > RFEs: > > 6341123 zpool create should allow using wildcards > 4868036 zpool create should have better support for total world domination > > The controller-based syntax has also been proposed before. > > - Eric > > -- > Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock-- Erik Trimble Java System Support Mailstop: usca14-102 Phone: x17195 Santa Clara, CA Timezone: US/Pacific (GMT-0800)
On Tue, Apr 04, 2006 at 12:51:34PM -0700, Erik Trimble wrote:> > > Onto my second inquiry (at the bottom of the previous note): > > can you add drives to a sub-device of a pool? That is, if a ZFS pool is > made up of udevs which are RAIDZ or mirrors, can one add disks to those > udevs without having to recreate the whole pool?You can add entire toplevel vdevs to any pool. For example, you can add another 5-way RAID-Z vdev to an existing pool. You can expand an existing mirror using ''zpool attach''. You cannot expand an existing RAID-Z vdev. While we have some ideas on the subject, it''s somewhat... hard. - Eric -- Eric Schrock, Solaris Kernel Development http://blogs.sun.com/eschrock
On Tue, 2006-04-04 at 17:50, Eric Schrock wrote:> Ahhhh! Just say no to metassist! If our interface isn''t > straightforward enough then something is wrong.It''s straightforward enough, but doesn''t have any support for policy or QOS constraints at all.> See the following > RFEs: > > 6341123 zpool create should allow using wildcards > 4868036 zpool create should have better support for total world dominationIf only. "see comments" doesn''t tell me very much... The related one, 4854068, does sound like what I''m after. It isn''t at all clear to me that adding this sort of higher level functionality directly into zpool create is necessarily a good thing. I can imagine it cluttering up and confusing the main zpool interface, and I don''t have a problem separating policy and implementation into separate commands. (Although I guess it could be a distinct subcommand.) -- -Peter Tribble L.I.S., University of Hertfordshire - http://www.herts.ac.uk/ http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/