I am in the beginning stage of converting multiple two-drive NAS devices to a
more proper single-device storage solution for my home network.
Because I have a pretty good understanding of hardware-based storage solutions,
originally I was going to go with a a traditional server-class motherboard that
allows for ecc memory, a depressingly expensive raid card that supports 8-12
drives and allowed for raid6, OCE (online capacity expansion), and spin down of
unused drives, and driven with FreeNAS.
A few months ago I happened upon ZFS and have been excitedly trying to learn all
I can about it. There is much to admire about ZFS, so I would like to integrate
it into my solution. The simple statement of requirements is: support for total
of 8-12 SATA2 hard drives, protection against data loss/corruption, and
protection against drive failure. Must have a mostly GUI interface
(regrettably, my command-line confidence is poor). Prefer to have a single large
storage pool. The ability to start with few drives and add storage as needed is
greatly desired.*
[ *if it is more future-proof, I could dig up some lower capacity drives and
start out with (for example) two 1.5tb drives and six 250gb drives with the plan
to replace a 250gb drive with a 1.5tb drive as needed]
When I feel I have a secure foothold of understanding I have little (if any)
trouble executing a plan. But I seem to be hung up on a multiple simple
questions that are of the "duh" variety for which I have been unable
to google myself to a solution. I normally take pride in finding my own
solutions so it is with some discomfort that I admit that I find myself in need
of a avuncular mentor to help me fit prior hardware raid knowledge with ZFS. If
I had to guess, I''d say that a ten minute verbal conversation combined
with maybe a dozen afterthought-induced questions that could be answered by
yes/no would get me to that secure foothold place.
A few examples of "duh" ?s
- How can I effect OCE with ZFS? The traditional ''back up all the
data somewhere, add a drive, re-establish the file system/pools/whatever, then
copy the data back'' is not going to work because there will be nowhere
to temporarily ''put'' the data.
- Concordantly, Is ZFS affected by a RAID card that supports OCE? Or is
this to no advantage?
- RAID5/6 with ZFS: As I understand it, ZFS with raidz will provide the
data/drive redundancy I seek [home network, with maybe two simultaneous users on
at least a P4 at 1GHz/1Gb RAM storage server] so obtaining a RAID controller
card is unnecessary/unhelpful. Yes?
--
This message posted from opensolaris.org
I consider myself also as a "noob" when it gets to ZFS but I already built myself a ZFS filer and maybe I can enlighten you by sharing my "advanced noob who read about a lot about ZFS" thoughts about ZFS> A few examples of "duh" ?s > - How can I effect OCE with ZFS? The traditional ''back up all the > data somewhere, add a drive, re-establish the file > system/pools/whatever, then copy the data back'' is not going to work > because there will be nowhere to temporarily ''put'' the data.RAIDZ/ZS expansion is a future target of the ZFS development but until today I was not able to determine when this will be available. Due to this fact I decided to assemble a pool with multiple mirrors. I did not want to end up with a 100% full RAIDZ and no ability to extend the pool with another drive and have to wait until RAIDZ expansion is "pruduction ready". Ok, you losse half the capacity of the vdefs but IO performance is way better than RAIDZ. It is possible to extend a existing RAIDZ vpool with another RAIDZ (or a mirror if you want, but I think this is not ideal because the reliability/performance of a zpool is also as good as the weakest item in the chain) but you might have to buy at least 3 drives to extend the vpool. Adding another mirror to the vpool extends performance as well as capacity which is not always true for RAIDZ (in the performance kind of view). For bulk storage IO performance is not that important...depends on your workload (YMMW, as always ;-)> - Concordantly, Is ZFS affected by a RAID card that supports OCE? > Or is this to no advantage?While RAID cards are supportet it is wiser to obtain a card which is able to publish the drives as JBOD and let ZFS handle the redundancy to benefit oft he ZFS checksumming. If ZFS can only see one device (whitch is redundant within the controller point of view) you might end up loosing some data when ZFS encounters a checksum error. I have a two way mirror consisting of 4 Seagate enterprise-type sata discs and doing a "zpool scrub" (aka. checksum check) once in a while. Even with those enterprise grade harddisks I encounter some inconsistencies on the drives. I am aware that this should not happen (ideally there are no errors reported on zppol scrub) but I i had no possibility to track down the source of error (all disks had errors until now, so I suppose the error is caused by the mainboard/cabling/controller)....but all errors until now could be repaired so this is n obig deal for me. Errors can only be corrected if ZFS handles the redundancy so it can fetch the failed blocks from another vdef.> - RAID5/6 with ZFS: As I understand it, ZFS with raidz will provide > the data/drive redundancy I seek [home network, with maybe two > simultaneous users on at least a P4 at 1GHz/1Gb RAM storage server] so > obtaining a RAID controller card is unnecessary/unhelpful. Yes?Yes! ;-) If only two users are accessing the ZFS pool simultaneously, RAIDZ would be sufficient for your type of use but I would go with dual redundancy (RAIDZ2) because if one harddisk dies and you do a resilver (rebuild the RAID) and one harddisk encounters chekcsum errors, you will loose data (ZFS will tell you which files are in question but I suppose you do not want to loose data at all) Read http://www.enterprisestorageforum.com/article.php/3839636 for more info about that. I would advise you to choose a mainboard with with at least 2 Gig of ECC-RAM because (IIRC) 1 Gig of RAM it the lowest suppoorted amount of RAM.> -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Fri, October 23, 2009 09:57, Robert wrote:> A few months ago I happened upon ZFS and have been excitedly trying to > learn all I can about it. There is much to admire about ZFS, so I would > like to integrate it into my solution. The simple statement of > requirements is: support for total of 8-12 SATA2 hard drives, protection > against data loss/corruption, and protection against drive failure. Must > have a mostly GUI interface (regrettably, my command-line confidence is > poor). Prefer to have a single large storage pool. The ability to start > with few drives and add storage as needed is greatly desired.* > [ *if it is more future-proof, I could dig up some lower capacity drives > and start out with (for example) two 1.5tb drives and six 250gb drives > with the plan to replace a 250gb drive with a 1.5tb drive as needed]Solaris is really pretty command-line oriented, and the people who use it are, so even if a GUI tool exists (sometimes it does), people tend to tell you how to do things from the command line. This generally works better for remote servers, and for people supporting dozens of servers.> When I feel I have a secure foothold of understanding I have little (if > any) trouble executing a plan. But I seem to be hung up on a multiple > simple questions that are of the "duh" variety for which I have been > unable to google myself to a solution. I normally take pride in finding my > own solutions so it is with some discomfort that I admit that I find > myself in need of a avuncular mentor to help me fit prior hardware raid > knowledge with ZFS. If I had to guess, I''d say that a ten minute verbal > conversation combined with maybe a dozen afterthought-induced questions > that could be answered by yes/no would get me to that secure foothold > place.You don''t happen to be around Minneapolis MN, do you? But really, chatting with me we''d both just end up with new and more interesting questions, I''m afraid. I''ve been running my home NAS on ZFS for something over a year now with good success (no data loss, though I did have to restore from backup at one point in an upgrade brangle). Sounds smaller than what you need, currently 800GB usable (2 vdevs each a 2-way mirror of 400GB disks).> A few examples of "duh" ?s > - How can I effect OCE with ZFS? The traditional ''back up all the data > somewhere, add a drive, re-establish the file system/pools/whatever, > then copy the data back'' is not going to work because there will be > nowhere to temporarily ''put'' the data.With ZFS currently, you can add a new VDEV into a pool, and the additional space is immediately available to everything (filesystems, volumes, etc.) allocated out of that pool. Also, mirrors and RAIDZ groups can expand to the size of the smallest device in the group -- so if you have a mirror of 2 400GB drives, you can replace one with something bigger, wait for resilver to complete (and be vulnerable to single failure during that period), then replace the other, resilver again (vulnerable to single failure during that period), and then the VDEV will have the new bigger size and the pool it''s in will have access to that space. Same thing for RAIDZ though with more drives hence more steps. For the mirror, by temporarily attaching a third drive you can avoid the window of vulnerability to single-disk failure (attach third large drive, wait for resilver, detach one small drive, attach another large drive, wait for resilver, detach remaining small drive, for example). What you CANNOT do yet is add an additional disk to a RAIDZ.> - Concordantly, Is ZFS affected by a RAID card that supports OCE? Or > is this to no advantage?For the kind of thing we''re doing, my impression is that fancy controller cards are a big-time losing proposition. They make management much more complex, prevent ZFS from doing its job, often lie about data security issues. And, as you say, cost money. Which doesn''t answer you question. I don''t know the answer to your question; I''ve avoided working with such cards with ZFS.> - RAID5/6 with ZFS: As I understand it, ZFS with raidz will provide the > data/drive redundancy I seek [home network, with maybe two simultaneous > users on at least a P4 at 1GHz/1Gb RAM storage server] so obtaining a RAID > controller card is unnecessary/unhelpful. Yes?Yes. ZFS does a good job with redundancy. Mirroring gives you 100% or higher (you can define 3-way, 4-way, 5-way, etc. mirrors to whatever absurd degree you want!) redundancy; RAIDZ gives you one redundant disk in the VDEV. RAIDZ2 gives you two redundant disks in the VDEV. You can pick the degree of redundancy that meets your needs. Performance will be a total non-issue with such a small user population. If they''re already used to the performance of getting files over the network, ZFS will be fine, may even improved perceived performance at the user workstation. Furthermore, the ZFS "scrub" operation and the fact that ZFS stores block checksums for data blocks (as well as metadata blocks) means you have more reason than with most home NAS setups to believe that the data on the disks is valid. Also, snapshots are a marvelous way to help people recover from fumble-fingered file deletions. I decided to use mirror pairs rather than RAIDZ groups for my vdevs at home. My hardware has 8 hot-swap bays but only 6 controllers, and two bays and two controllers are used for the boot zpool, so I currently only have 4 slots for data drives. With another controller I could bring that up to 6 slots. My thinking was that if I used all 4 slots for a raidz vdev, I''d get more storage, but when I needed to upgrade it I''d have to replace all 4 drives to get an increase. So I started with a single mirror of 400GB disks. That lasted me 6 months or so; I then upgraded by adding a second mirror of 400GB disks to the zpool (I used the same size because I''d lucked into a batch of free 400GB disks some time ago). Now, all my slots are in use. When I need to add space, I''ll need to replace existing drives. But I can do that in pairs, I don''t need to replace all 4 at once. So, next time around, perhaps in 6 months, I''ll replace two 400GB disks with 1.5TB disks and gain over 1TB usable space. I can do that again with the other mirror pair, when needed. And when I''ve used up THAT there will probably be disks bigger than 1TB available (since 2TB is available today), or else by then it''ll be time to replace the whole platform, one or the other. Mirrors also give significantly increased read performance over RAIDZ, but I can''t see that being significant for a home network situation like ours. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
> - How can I effect OCE with ZFS? The traditional > ''back up all the data somewhere, add a drive, > re-establish the file system/pools/whatever, then > copy the data back'' is not going to work because > there will be nowhere to temporarily ''put'' the > data.Add devices to the pool. Preferably in mirrors or raidz configurations. If you just add bare devices to the pool you are running RAID-0, no redundancy. You cannot add devices to a raidz, as mentioned. But you can add more raidz or mirror devices. You can also replace devices with larger ones. It would be nice to be able to add more devices to a raidz for home users like us, maybe we''ll see it someday. For now, the capabilities we do have make it reasonable to deal with.> - Concordantly, Is ZFS affected by a RAID card > that supports OCE? Or is this to no advantage?Don''t bother. Spend the money on more RAM, and drives. :) Do get a nice controller though. Supermicro makes a few nice units. I''m using 2 AOC-USAS-L8i cards. They work great, though you do have to mod the mounting bracket to get them to work in a standard case. These are based on LSI cards, I just found them cheaper than the same LSI branded card. Avoid the cheap $20 4-port jobs. I''ve had a couple of them die already. Thankfully, I didn''t lose any data... I think... no ZFS on that box.> - RAID5/6 with ZFS: As I understand it, ZFS with > raidz will provide the data/drive redundancy I seek > [home network, with maybe two simultaneous users on > at least a P4 at 1GHz/1Gb RAM storage server] so > obtaining a RAID controller card is > unnecessary/unhelpful. Yes?Correct. Though I would increase the RAM personally, it''s so cheap these days. My home fileserver has 8GB of ECC RAM. I''m also running Xen VMs though, so some of my RAM is used for running those. You can even do tripple-redundant raidz with ZFS now, so you could lose 3 drives without any data loss. For those that want really high availability, or really big arrays I suppose. I''m running 4x1.5TB in a raidz1, no problems. I do plan to keep a spare around though. I''ll just use it to store backups to start with. If a drive goes bad, I''ll drop it in and do a zpool replace. Don''t worry about the command line. The ZFS based commands are pretty short and simple. Read up on zpool and zfs. Those are the commands you use the most for managing ZFS. There''s also the ZFS best practices guide if you haven''t seen it. Useful advice in there. -- This message posted from opensolaris.org
You dont need a hw raid card with ZFS, instead ZFS prefers to work alone. It is the best solution to ditch a hw raid card. I strongly advice you to use raidz2 (raid-6). Because if you use raidz1 (raid-5) and a drive fails, you have to swap that disc and repair your zfs raid. That will cause lots of stress on your other drives. If you try to repair a 1TB drive, it can take 6h. If you try to repair a >4TB it can take a week. During that time, if another drive fails, your data is lost. It takes long time to repair a raid, because the transfer speeds are slow compared to storage capacity. If you have lots of discs, it is adviced to create several groups (vdev) and make one large zpool of the groups. Each group should be a raidz1 or raidz2. I prefer raidz2. You can add a new group later if you wish. But you can not remove a group. But, it is as easy to just swap the drives to larger ones. Say you have 1TB drives in your zpool, then you can swap one drive at a time to 2TB drives. Then your zpool have grown in capacity. Therefore I have a zpool of 8 drives 1TB. If I need more capacity, I just swap them to 2TB or 4TB drives. Instead of adding more and more small drives that consume power and make noice. In the future, one drive will be 10TB. Then I regret I have lots of small drives. Therefore, I would make a zpool of 8 discs, configured as one raidz2. And add no more drives. Instead, I would swap the drives to larger and larger. If I had 16 drives, I would make two raidz2 groups. One raidz2 should consist of 6-8 drives. It is better to have several smaller groups in your zpool, than one large group - because each group increases performance. (Not sure on this: It is similar to one group gives you the performance of ONE drive. If you have 24 discs in one group, they will be slow as one drive, when talking about IOPS. If you have 8 discs in 3 groups, they will be fast as three drives, when talking about IOPS, i.e. head move speed and latency) if you mix sizes of discs, only the smallest size will be valid in one group. Say you have 500GB and 1TB drives. If you make one group, it will 500GB drives. The proper way would be to make two different groups (each one raidz1/raidz2), and add them to your zpool. Lastly, you must read ALL of these articles: http://breden.org.uk/2008/03/02/a-home-fileserver-using-zfs/ -- This message posted from opensolaris.org