Hi, (Main questions at bottom of post) I recently discovered the joys of ZFS. I have a home file server for backups+media, also hosting some virtual machine (over LAN). I was wondering if I could get some feedback as to whether I have set things up properly. Drives: 20 x 1TB (Mix of seagate and Hitachi) 4 x 2TB (WD20EARS) Also the server has 24GB ram (for the virtual machines, but I can leave lots free for ZFS) I deleted the partitions off of all drives. Then created 4 vdevs each with 5 1TB drives in raidz. I get some errors at bootup about no valid partition table or soemthing on the drives, is this to be expected? Now I would like to add my 4 x 2TB drives, I get a warning message saying that: "Pool uses 5-way raidz and new vdev uses 4-way raidz" Do you think it would be safe to use the -f switch here? My chassis is at its limit in terms of hdd, 24(hot swap bays).. So I would have to stick one inside somewhere.. In case I lost you guys up there, I have 2 main questions: 1 - Errors in dmesg while booting about "Corrupt of Invalid GPT detected" "The secodary GPT is corrupt or invalid" This can be ignored as the drives giving this error are all part of the pool. 2 - I would like to add a 4 drive raidz to my pool, although zfs warns that my current vdevs are all 5-way raidz.. Is it safe (and recommended ) to use the -f switch and add a 4x2TB raidz vdev to the pool? Thanks! -- This message posted from opensolaris.org
On Tue, 13 Apr 2010, Christian Molson wrote:> > Now I would like to add my 4 x 2TB drives, I get a warning message > saying that: "Pool uses 5-way raidz and new vdev uses 4-way raidz" > Do you think it would be safe to use the -f switch here?It should be "safe" but chances are that your new 2TB disks are considerably slower than the 1TB disks you already have. This should be as much cause for concern (or more so) than the difference in raidz topology. Maybe you should create a second temporary pool with these drives and exercise them under load for a while to see how they behave. If they behave well, then destroy the temporary pool and add the drives to your main pool. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Tue, April 13, 2010 09:48, Christian Molson wrote:> > Now I would like to add my 4 x 2TB drives, I get a warning message saying > that: "Pool uses 5-way raidz and new vdev uses 4-way raidz" Do you think > it would be safe to use the -f switch here?Yes. 4-way on the bigger drive is *more* redundancy (25%, rather than 20%) (though not necessarily "safer", since the bigger drive increases recovery time) than 5-way on the smaller drive. I''d describe these as "vaguely" the same level of redundancy, and hence not especially inappropriate to put in the same pool. Putting a single disk into a pool that''s otherwise RAIDZ would be a bad idea, obviously, and that''s what that message is particularly to warn you about I believe. However, I have some doubts about using 2TB drives with single redundancy in general. It takes a LONG time to resilver a drive that big, and during the resilver you have no redundancy and are hence subject to data loss if one of the remaining drives also fails. And resilvering puts extra stress on the IO system and drives, so probably the risk of failure is increased. (If your backups are good enough, you may plan to cover the possibility of that second failure by restoring from backups. That works, if they''re really good enough; it just takes more work and time.) 24 hot-swap bays in your home chassis? Now that does sound pretty extreme. I felt like my 8-bay chassis is a bit excessive for home; and it only has 6 bays populated with data-disks, and they''re just 400GB. And I store a lot of RAW files from DSLRs on it it, I feel like I use quite a bit of space (until I see somebody come along casually talking about vaguely 10 times more space). How DO you deal with backup at that data size? I can back up to a single external USB disk (I have 3 I rotate), and a full backup completes overnight. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
On Tue, April 13, 2010 10:38, Bob Friesenhahn wrote:> On Tue, 13 Apr 2010, Christian Molson wrote: >> >> Now I would like to add my 4 x 2TB drives, I get a warning message >> saying that: "Pool uses 5-way raidz and new vdev uses 4-way raidz" >> Do you think it would be safe to use the -f switch here? > > It should be "safe" but chances are that your new 2TB disks are > considerably slower than the 1TB disks you already have. This should > be as much cause for concern (or more so) than the difference in raidz > topology.Not necessarily for a home server. While mine so far is all mirrored pairs of 400GB disks, I don''t even think about "performance" issues, I never come anywhere near the limits of the hardware. Your suggestion (snipped) that he test performance on the new drives to see how they differ is certainly good if he needs to worry about performance. Testing actual performance in your own exact hardware is always smart. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
On Wed, 14 Apr 2010, David Dyer-Bennet wrote:>> It should be "safe" but chances are that your new 2TB disks are >> considerably slower than the 1TB disks you already have. This should >> be as much cause for concern (or more so) than the difference in raidz >> topology. > > Not necessarily for a home server. While mine so far is all mirrored > pairs of 400GB disks, I don''t even think about "performance" issues, I > never come anywhere near the limits of the hardware.I don''t see how the location of the server has any bearing on required performance. If these 2TB drives are the new 4K sector variety, even you might notice. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Wed, April 14, 2010 12:06, Bob Friesenhahn wrote:> On Wed, 14 Apr 2010, David Dyer-Bennet wrote: >>> It should be "safe" but chances are that your new 2TB disks are >>> considerably slower than the 1TB disks you already have. This should >>> be as much cause for concern (or more so) than the difference in raidz >>> topology. >> >> Not necessarily for a home server. While mine so far is all mirrored >> pairs of 400GB disks, I don''t even think about "performance" issues, I >> never come anywhere near the limits of the hardware. > > I don''t see how the location of the server has any bearing on required > performance. If these 2TB drives are the new 4K sector variety, even > you might notice.The location does not, directly, of course; but the amount and type of work being supported does, and most home servers see request streams very different from commercial servers. The last server software I worked on was able to support 80,000 simultaneous HD video streams. Coming off Thumpers, in fact (well, coming out of a truly obscene amount of DRAM buffer on the streaming board, which was in turn loaded from Thumpers); this was the thing that Thumper was originally designed for, known when I worked there as the Sun Streaming System I believe. You don''t see loads like that on home servers :-). And a big database server would have an equally extreme but totally different access pattern. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
On Wed, 14 Apr 2010, David Dyer-Bennet wrote:>>> >>> Not necessarily for a home server. While mine so far is all mirrored >>> pairs of 400GB disks, I don''t even think about "performance" issues, I >>> never come anywhere near the limits of the hardware. >> >> I don''t see how the location of the server has any bearing on required >> performance. If these 2TB drives are the new 4K sector variety, even >> you might notice. > > The location does not, directly, of course; but the amount and type of > work being supported does, and most home servers see request streams very > different from commercial servers.If it was not clear, the performance concern is primarily for writes since zfs will load-share the writes across the available vdevs using an algorithm which also considers the write queue/backlog for each vdev. If a vdev is slow, then it may be filled more slowly than the other vdevs. This is also the reason why zfs encourages that all vdevs use the same organization. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
On Wed, April 14, 2010 12:29, Bob Friesenhahn wrote:> On Wed, 14 Apr 2010, David Dyer-Bennet wrote: >>>> >>>> Not necessarily for a home server. While mine so far is all mirrored >>>> pairs of 400GB disks, I don''t even think about "performance" issues, I >>>> never come anywhere near the limits of the hardware. >>> >>> I don''t see how the location of the server has any bearing on required >>> performance. If these 2TB drives are the new 4K sector variety, even >>> you might notice. >> >> The location does not, directly, of course; but the amount and type of >> work being supported does, and most home servers see request streams >> very >> different from commercial servers. > > If it was not clear, the performance concern is primarily for writes > since zfs will load-share the writes across the available vdevs using > an algorithm which also considers the write queue/backlog for each > vdev. If a vdev is slow, then it may be filled more slowly than the > other vdevs. This is also the reason why zfs encourages that all > vdevs use the same organization.As I said, I don''t think of performance issues on mine. So I wasn''t thinking of that particular detail, and it''s good to call it out explicitly. If the performance of the new drives isn''t adequate, then the performance of the entire pool will become inadequate, it looks like. I expect it''s routine to have disks of different generations in the same pool at this point (and if it isn''t now, it will be in 5 years), just due to what''s available, replacing bad drives, and so forth. -- David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info
First I want to thank everyone for their input, It is greatly appreciated. To answer a few questions: Chassis I have: http://www.supermicro.com/products/chassis/4U/846/SC846E2-R900.cfm Motherboard: http://www.tyan.com/product_board_detail.aspx?pid=560 RAM: 24 GB (12 x 2GB) 10 x 1TB Seagates 7200.11 10 x 1TB Hitachi 4 x 2TB WD WD20EARS (4K blocks) I used to have (selling it now) a 3Ware 9690SA controller, and had setup rather poorly. I added drives to the RAID6 array, and then created new partitions on it which were concatenated via LVM. I ran EXT3 as a filesystem as well. Firstly, the 3Ware controller was ok, but the limitations with HW raid were what brought me to ZFS. Mainly the fact that you cannot shrink the size of a RAID6 HW array since it has no knowledge of the FS. Most of the VM''s and data files I store here are not critical. I make backups of the important stuff (family pictures, work etc). I also backup the data within the VM''s so if their disk files are ever lost it is not too much of a problem.>From what you guys have said, adding slow drives to the pool will cause them to be a bottleneck in the pool. I am just finishing up some copying etc, and will benchmark a test pool with the 2TB drives. Even if they are fast enough, I think It would be better to create a seperate pool for them, and store only data which can be lost.Also, do you guys have any suggestions for this: I have a desktop running windows 7, it runs off of an SSD. Would setting up NFS or iSCSI (even possible?) to install other apps on be worth it? I am guessing it would be easier to just pop a regular drive in it instead.. Thanks again! -- This message posted from opensolaris.org
Just a quick update, Tested using bonnie++ just during its "Intelligent write": my 5 vdevs of 4x1tb drives wrote around 300-350MB/sec using that test. The 1vdev of 4x2TB drives wrote more inconsistently, between 200-300. This is not a complete test... just looking at iostat output while bonnie++ ran.. I will do a complete test later on, but it seems initially that the new drives are not horrible in a pool of 4 raidz. -- This message posted from opensolaris.org
On 04/14/10 12:37, Christian Molson wrote:> First I want to thank everyone for their input, It is greatly appreciated. > > To answer a few questions: > > Chassis I have: http://www.supermicro.com/products/chassis/4U/846/SC846E2-R900.cfm > > Motherboard: > http://www.tyan.com/product_board_detail.aspx?pid=560 > > RAM: > 24 GB (12 x 2GB) > > 10 x 1TB Seagates 7200.11 > 10 x 1TB Hitachi > 4 x 2TB WD WD20EARS (4K blocks)If you have the spare change for it I''d add one or two SSD''s to the mix, with space on them allocated to the root pool plus l2arc cache, and slog for the data pool(s). - Bill