Hi all, I have a Thumper (ok, actually 3) with each having one large pool, multiple filesystems and many snapshots. They are holding rsync copies of multiple clients, being synced every night (using snapshots to keep ''incremental'' backups). I''m wondering how often (if ever) I should do scrubs of the pools, or if the internal zfs integrity is enough that I don''t need to do manual scrubs of the pool? I read through a number of tutorials online as well as the zfs wiki entry, but I didn''t see anything very pertinent. Scrubs are I/O intensive, but is the Pool able to be used normally during a scrub? I think the answer is yes, but some confirmation helps me sleep at night. Thoughts? Ideas? Knife-fights? Thanks Dave David Glaser Systems Administrator Senior LSA Information Technology University of Michigan -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081201/842ec8ac/attachment.html>
Hi, I scrub my pools once a week on the weekend, rpool saturday, cesspool (my other pool) Sunday. You can still use the box while it is scrubbing but as you would expect the I/O is very slow and can sometimes be close to unusable. Ta, --- Cooper Ry Lees UNIX Evangelist - Information Management Services (IMS) Australian Nuclear Science and Technology Organisation T +61 2 9717 3853 F +61 2 9717 9273 M +61 403 739 446 E cooper.lees at ansto.gov.au www.ansto.gov.au Important: This transmission is intended only for the use of the addressee. It is confidential and may contain privileged information or copyright material. If you are not the intended recipient, any use or further disclosure of this communication is strictly forbidden. If you have received this transmission in error, please notify me immediately by telephone and delete all copies of this transmission as well as any attachments. On 02/12/2008, at 2:05 PM, Glaser, David wrote:> Hi all, > > I have a Thumper (ok, actually 3) with each having one large pool, > multiple filesystems and many snapshots. They are holding rsync > copies of multiple clients, being synced every night (using > snapshots to keep ?incremental? backups). > > I?m wondering how often (if ever) I should do scrubs of the pools, > or if the internal zfs integrity is enough that I don?t need to do > manual scrubs of the pool? I read through a number of tutorials > online as well as the zfs wiki entry, but I didn?t see anything very > pertinent. Scrubs are I/O intensive, but is the Pool able to be used > normally during a scrub? I think the answer is yes, but some > confirmation helps me sleep at night. > > Thoughts? Ideas? Knife-fights? > > Thanks > Dave > > David Glaser > Systems Administrator Senior > LSA Information Technology > University of Michigan > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20081202/c3f81f04/attachment.html>
On 1-Dec-08, at 10:05 PM, Glaser, David wrote:> Hi all, > > > > I have a Thumper (ok, actually 3) with each having one large pool, > multiple filesystems and many snapshots. They are holding rsync > copies of multiple clients, being synced every night (using > snapshots to keep ?incremental? backups). > > > > I?m wondering how often (if ever) I should do scrubs of the pools, > or if the internal zfs integrity is enough that I don?t need to do > manual scrubs of the pool? >Yes you should. Passive integrity is not all; proactively reading the pool improves your MTTDL substantially, see other sources for the actual figures. :) It does not need to be very frequent. I do it monthly on my colo server. --Toby> I read through a number of tutorials online as well as the zfs wiki > entry, but I didn?t see anything very pertinent. Scrubs are I/O > intensive, but is the Pool able to be used normally during a scrub? > I think the answer is yes, but some confirmation helps me sleep at > night. > > > > Thoughts? Ideas? Knife-fights? > > > > Thanks > > Dave > > > > David Glaser > > Systems Administrator Senior > > LSA Information Technology > > University of Michigan > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Glaser, David wrote:> > Hi all, > > I have a Thumper (ok, actually 3) with each having one large pool, > multiple filesystems and many snapshots. They are holding rsync copies > of multiple clients, being synced every night (using snapshots to keep > ?incremental? backups). > > I?m wondering how often (if ever) I should do scrubs of the pools, or > if the internal zfs integrity is enough that I don?t need to do manual > scrubs of the pool? I read through a number of tutorials online as > well as the zfs wiki entry, but I didn?t see anything very pertinent. > Scrubs are I/O intensive, but is the Pool able to be used normally > during a scrub? I think the answer is yes, but some confirmation helps > me sleep at night. >We did a study on re-write scrubs which showed that once per year was a good interval for modern, enterprise-class disks. However, ZFS does a read-only scrub, so you might want to scrub more often.> Thoughts? Ideas? Knife-fights? >Knife fights? Naw, more like "paranoia will destroy ya" :-) Maybe we need a ZFS theme song :-) http://www.youtube.com/watch?v=g3OVaCDLc9M http://www.youtube.com/watch?v=ZBbAZVw3_7A -- richard
> I have a Thumper (ok, actually 3) with each having one large pool,multiple> filesystems and many snapshots. They are holding rsync copies ofmultiple> clients, being synced every night (using snapshots to keep''incremental''> backups). > > I''m wondering how often (if ever) I should do scrubs of the pools, orif> the internal zfs integrity is enough that I don''t need to do manualscrubs> of the pool? I read through a number of tutorials online as well asthe zfs> wiki entry, but I didn''t see anything very pertinent. Scrubs are I/O > intensive, but is the Pool able to be used normally during a scrub? I > think the answer is yes, but some confirmation helps me sleep atnight. Scrubs are the lowest priority, so I understand it should theoretically work fine. We''ve got two 48TB thumpers, with a nightly rsync from the main to the reserve. I''m currently running a scrub every Friday at 23:02, which last week took 5h15 to scrub the 7TB of used data (about 5TB of real, 2TB of snapshots) on the single pool. That''s about 380MBytes/second. -- "09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0" Paul Weaver Systems Development Engineer News Production Facilities, BBC News
Ok, thanks for all the responses. I''ll probably do every other week scrubs, as this is the backup data (so doesn''t need to be checked constantly). I''m a little concerned about the time involved to do 33TB (after the 48TB has been RAIDed fully) when it is fully populated with filesystems and snapshots, but I''ll keep an eye on it. Thanks all. Dave -----Original Message----- From: Paul Weaver [mailto:paul.weaver at bbc.co.uk] Sent: Tuesday, December 02, 2008 8:11 AM To: Glaser, David; zfs-discuss at opensolaris.org Subject: RE: [zfs-discuss] How often to scrub?> I have a Thumper (ok, actually 3) with each having one large pool,multiple> filesystems and many snapshots. They are holding rsync copies ofmultiple> clients, being synced every night (using snapshots to keep''incremental''> backups). > > I''m wondering how often (if ever) I should do scrubs of the pools, orif> the internal zfs integrity is enough that I don''t need to do manualscrubs> of the pool? I read through a number of tutorials online as well asthe zfs> wiki entry, but I didn''t see anything very pertinent. Scrubs are I/O > intensive, but is the Pool able to be used normally during a scrub? I > think the answer is yes, but some confirmation helps me sleep atnight. Scrubs are the lowest priority, so I understand it should theoretically work fine. We''ve got two 48TB thumpers, with a nightly rsync from the main to the reserve. I''m currently running a scrub every Friday at 23:02, which last week took 5h15 to scrub the 7TB of used data (about 5TB of real, 2TB of snapshots) on the single pool. That''s about 380MBytes/second. -- "09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0" Paul Weaver Systems Development Engineer News Production Facilities, BBC News
So you''ve got a zpool across 46 (48?) of the disks? When I was looking into our thumpers everyone seemed to think a raidz over more than 10 disks was a hideous idea. -- Paul Weaver Systems Development Engineer News Production Facilities, BBC News Work: 020 822 58109 Room 1244 Television Centre, Wood Lane, London, W12 7RJ> -----Original Message----- > From: zfs-discuss-bounces at opensolaris.org > [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of > Glaser, David > Sent: 02 December 2008 13:24 > To: zfs-discuss at opensolaris.org > Subject: Re: [zfs-discuss] How often to scrub? > > Ok, thanks for all the responses. I''ll probably do every > other week scrubs, as this is the backup data (so doesn''t > need to be checked constantly). I''m a little concerned about > the time involved to do 33TB (after the 48TB has been RAIDed > fully) when it is fully populated with filesystems and > snapshots, but I''ll keep an eye on it. > > Thanks all. > > Dave > > > -----Original Message----- > From: Paul Weaver [mailto:paul.weaver at bbc.co.uk] > Sent: Tuesday, December 02, 2008 8:11 AM > To: Glaser, David; zfs-discuss at opensolaris.org > Subject: RE: [zfs-discuss] How often to scrub? > > > I have a Thumper (ok, actually 3) with each having one large pool, > multiple > > filesystems and many snapshots. They are holding rsync copies of > multiple > > clients, being synced every night (using snapshots to keep > ''incremental'' > > backups). > > > > I''m wondering how often (if ever) I should do scrubs of the > pools, or > if > > the internal zfs integrity is enough that I don''t need to do manual > scrubs > > of the pool? I read through a number of tutorials online as well as > the zfs > > wiki entry, but I didn''t see anything very pertinent. > Scrubs are I/O > > intensive, but is the Pool able to be used normally during > a scrub? I > > think the answer is yes, but some confirmation helps me sleep at > night. > > Scrubs are the lowest priority, so I understand it should > theoretically work fine. > > We''ve got two 48TB thumpers, with a nightly rsync from the > main to the reserve. I''m currently running a scrub every > Friday at 23:02, which last week took 5h15 to scrub the 7TB > of used data (about 5TB of real, 2TB of > snapshots) on the single pool. That''s about 380MBytes/second. > > > -- > > "09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0" > > Paul Weaver > Systems Development Engineer > News Production Facilities, BBC News > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
On Tue, Dec 2, 2008 at 10:15, Paul Weaver <paul.weaver at bbc.co.uk> wrote:> So you''ve got a zpool across 46 (48?) of the disks? > > When I was looking into our thumpers everyone seemed to think a raidz > over > more than 10 disks was a hideous idea.A vdev that size is bad, a pool that size composed of multiple vdevs is fine. Raidz2 is always recommended over raidz. Will
On 2-Dec-08, at 8:24 AM, Glaser, David wrote:> Ok, thanks for all the responses. I''ll probably do every other week > scrubs, as this is the backup data (so doesn''t need to be checked > constantly).Even that is probably more frequent than necessary. I''m sure somebody has done the MTTDL math. IIRC, the big win is doing any scrubbing at all. The difference between scrubbing every 2 weeks and every 2 months may be negligible. (IANAMathematician tho) --T> I''m a little concerned about the time involved to do 33TB (after > the 48TB has been RAIDed fully) when it is fully populated with > filesystems and snapshots, but I''ll keep an eye on it. > > Thanks all. > > Dave > > > -----Original Message----- > From: Paul Weaver [mailto:paul.weaver at bbc.co.uk] > Sent: Tuesday, December 02, 2008 8:11 AM > To: Glaser, David; zfs-discuss at opensolaris.org > Subject: RE: [zfs-discuss] How often to scrub? > >> I have a Thumper (ok, actually 3) with each having one large pool, > multiple >> filesystems and many snapshots. They are holding rsync copies of > multiple >> clients, being synced every night (using snapshots to keep > ''incremental'' >> backups). >> >> I''m wondering how often (if ever) I should do scrubs of the pools, or > if >> the internal zfs integrity is enough that I don''t need to do manual > scrubs >> of the pool? I read through a number of tutorials online as well as > the zfs >> wiki entry, but I didn''t see anything very pertinent. Scrubs are I/O >> intensive, but is the Pool able to be used normally during a scrub? I >> think the answer is yes, but some confirmation helps me sleep at > night. > > Scrubs are the lowest priority, so I understand it should > theoretically > work fine. > > We''ve got two 48TB thumpers, with a nightly rsync from the main to the > reserve. I''m currently running a scrub every Friday at 23:02, which > last > week took 5h15 to scrub the 7TB of used data (about 5TB of real, > 2TB of > snapshots) on the single pool. That''s about 380MBytes/second. > > > -- > > "09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0" > > Paul Weaver > Systems Development Engineer > News Production Facilities, BBC News > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Tue, 2 Dec 2008, Toby Thain wrote:> > Even that is probably more frequent than necessary. I''m sure somebody > has done the MTTDL math. IIRC, the big win is doing any scrubbing at > all. The difference between scrubbing every 2 weeks and every 2 > months may be negligible. (IANAMathematician tho)This surely depends on the type of hardware used. If the disks are not true "enterprise" grade (e.g. ordinary SATA drives) then scrubbing more often is likely warranted since these are much more likely to exhibit user-visible decay over a period of time and the scrub will find (and correct) the decaying bits before it is too late. The enterprise-class disks should not require scrubbing very often. Enterprise disks may be almost as likely to go entirely belly up as they are to produce a bad sector. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/