Hi all! I''ve decided to take the "big jump" and build a ZFS home filer (although it might also do "other work" like caching DNS, mail, usenet, bittorent and so forth). YAY! I wonder if anyone can shed some light on how long a pool scrub would take on a fairly decent rig. These are the specs as-ordered: Asus P5Q-EM mainboard Core2 Quad 2.83 GHZ 8GB DDR2/80 OS: 2 x SSD''s in RAID 0 (brand/size not decided on yet, but they will definitely be some flavor of SSD) Data: 4 x 1TB Samsung Spin Point 7200 RPM 32MB cache SATA HD''s (RAIDZ) Data payload initially will be around 550GB or so, (before loading any stuff from another NAS and so on) Does scrub like memory, or CPU, or both? There''s enough horsepower available, I would think. Same question applies to resilvering if I need to swap out drives at some point. [cough] I can''t wait to get this thing built! :) Regards & TIA, Your just-subscribed total ZFS n00b, -Me -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091122/3b768589/attachment.html>
On Sun, Nov 22, 2009 at 10:15 AM, Colin Raven <colin at clearcutnetworks.com>wrote:> Hi all! > I''ve decided to take the "big jump" and build a ZFS home filer (although it > might also do "other work" like caching DNS, mail, usenet, bittorent and so > forth). YAY! I wonder if anyone can shed some light on how long a pool scrub > would take on a fairly decent rig. These are the specs as-ordered: > > Asus P5Q-EM mainboard > Core2 Quad 2.83 GHZ > 8GB DDR2/80 > > OS: > 2 x SSD''s in RAID 0 (brand/size not decided on yet, but they will > definitely be some flavor of SSD) > > Data: > 4 x 1TB Samsung Spin Point 7200 RPM 32MB cache SATA HD''s (RAIDZ) > > Data payload initially will be around 550GB or so, (before loading any > stuff from another NAS and so on) > > Does scrub like memory, or CPU, or both? There''s enough horsepower > available, I would think. Same question applies to resilvering if I need to > swap out drives at some point. [cough] I can''t wait to get this thing built! > :) > > Regards & TIA, > > Your just-subscribed total ZFS n00b, > -Me > >The OS doesn''t really have a need for ssd''s. You''d be better off getting a large SATA drive for your OS and using the ssd''s as readzilla/logzilla. I think everyone will also tell you to get ECC ram. To do this cheaply, it generally means going the AMD route. -- --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20091122/56fb10a4/attachment.html>
Colin Raven wrote:> Hi all! > I''ve decided to take the "big jump" and build a ZFS home filer > (although it might also do "other work" like caching DNS, mail, > usenet, bittorent and so forth). YAY! I wonder if anyone can shed some > light on how long a pool scrub would take on a fairly decent rig. > These are the specs as-ordered: > > Asus P5Q-EM mainboard > Core2 Quad 2.83 GHZ > 8GB DDR2/80On this system (Dual core Athlon64 4600+ 2.4GHz, 8GB ram), 7200RPM enterprise duty disks, it takes about 14 hours on the following zpool: # zpool status pool: export state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using ''zpool upgrade''. Once this is done, the pool will no longer be accessible on older software versions. scrub: scrub completed after 13h45m with 0 errors on Fri Nov 20 15:05:20 2009 config: NAME STATE READ WRITE CKSUM export ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c3d0 ONLINE 0 0 0 c2d0 ONLINE 0 0 0 errors: No known data errors # zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT export 930G 503G 427G 54% ONLINE - #> OS: > 2 x SSD''s in RAID 0 (brand/size not decided on yet, but they will > definitely be some flavor of SSD)Can only be RAID1 (mirrored) for the OS.> Data: > 4 x 1TB Samsung Spin Point 7200 RPM 32MB cache SATA HD''s (RAIDZ)Prefer mirrors myself, but depends on how much data you have verses how much disk you can afford.> Data payload initially will be around 550GB or so, (before loading any > stuff from another NAS and so on) > > Does scrub like memory, or CPU, or both? There''s enough horsepower > available, I would think. Same question applies to resilvering if I > need to swap out drives at some point. [cough] I can''t wait to get > this thing built! :)I also use the system as a desktop now (after doing some system consolidation). Scrub doesn''t use much CPU, but it does interfere with the interactive response of the desktop. I suspect this is due to the i/o''s it queues up on the disks. -- Andrew
Yesterday''s integration of 6678033 resilver code should prefetch as part of changeset 74e8c05021f1 (which should be in build 129 when it comes out) may improve scrub times, particularly if you have a large number of small files and a large number of snapshots. I recently tested an early version of the fix, and saw one pool go from an elapsed time of 85 hours to 20 hours; another (with many fewer snapshots) went from 35 to 17. - Bill
On my home server (currently having problems with random reboots), it takes around 1.5hrs to do a scrub of my RAIDZ1 6 x 1.5TB array, with around 2TB of data on it. Specs are: CPU: core2duo 2.5GHz RAM: 2GB 800Mhz DDR2 OS DIsks: 120GB Seagate ATA Storage drives: 6 x 15TB seagate sata2 7200rpm -- This message posted from opensolaris.org
Bill Sommerfeld wrote:> Yesterday''s integration of > > 6678033 resilver code should prefetch > > as part of changeset 74e8c05021f1 (which should be in build 129 when it > comes out) may improve scrub times, particularly if you have a large > number of small files and a large number of snapshots. I recently > tested an early version of the fix, and saw one pool go from an elapsed > time of 85 hours to 20 hours; another (with many fewer snapshots) went > from 35 to 17. >I''ve been wondering what difference that might make. I''m currently running snv_125. $ zfs list -t snapshot | wc -l 4407 $ Yep, quite a few snapshots. However, more important to me would be reducing the impact a scrub has on the rest of the system, even if it takes longer. Conversely, I can imagine you might want a resilver to run as fast as possible in some cases, even at the expense of other system activities. It would be really nice to have a speed knob on these operations, which you can vary while the activity progresses, depending on other uses of the system. -- Andrew
Colin Raven wrote:> Asus P5Q-EM mainboard > Core2 Quad 2.83 GHZ > 8GB DDR2/80 > > OS: > 2 x SSD''s in RAID 0 (brand/size not decided on yet, but they will > definitely be some flavor of SSD) > > Data: > 4 x 1TB Samsung Spin Point 7200 RPM 32MB cache SATA HD''s (RAIDZ)as i have a quite comparable setup: my HW: ASUS P5Q-EM Core 2 Duo E8400 (2x3GHz) 4GB DDR/800 OS: 1x 250GB ATA (old&dusty, thinking about to replace this with an SSD) Data: 4x 1TB Seagate Pipeline HD (2x533CS, 2x322CS) NAME SIZE USED AVAIL CAP HEALTH ALTROOT pool 3.64T 943G 2.72T 25% ONLINE - Status: pool: pool state: ONLINE scrub: scrub completed after 1h25m with 0 errors on Wed Dec 2 16:43:35 2009 config: NAME STATE READ WRITE CKSUM pool ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 c1t5d0 ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 c1t1d0 ONLINE 0 0 0 errors: No known data errors resilvering a drive took about 1:30h to 2:15h (i upgraded the pool from 500Gig drives to the 1TB drives) HTH, Stpehan
sorry to bring the old one up, but I think it is better than make a new one ?? Are there some one who have some resilver time from a raidz1/2 pool whith 5TB+ data on it ? -- This message posted from opensolaris.org
Hi On Sunday 19 December 2010 11:12:32 Tobias Lauridsen wrote:> sorry to bring the old one up, but I think it is better than make a new one > ?? Are there some one who have some resilver time from a raidz1/2 pool > whith 5TB+ data on it ?if you just looked into the discussion over the past day (or week), you would learn that the resilver time depends on the amount of writes to the system while resilvering. On an idle system you might be able to guesstimate this by taking the disk size and the number of iops of the disk and the system into account, usually a couple of hours should be alright. Cheers Carsten
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Tobias Lauridsen > > sorry to bring the old one up, but I think it is better than make a newone ??> Are there some one who have some resilver time from a raidz1/2 pool whith > 5TB+ data on it ?resilver & scrub time aren''t primarily influenced by the number of bytes to resilver or scrub. It''s primarily influenced by the number of fragments in said data... If you have no snapshots, and you never did, and if your files were initially written serially and quickly... Then your scrub & resilver time should be very quick, because all your data is laid out on disk serially and therefore you''re able to cover a large number of GB/min. But if you have been performing random reads/writes over a long period of time... creating and destroying snapshots... etc ... Then it could be pretty awful. All of this is, of course, no concern for SSD drives with really high IOPS.