I've got a 9 sata drive raidz1 array, started at version 6, upgraded to version 13. I had an apparent drive failure, and then at some point, a kernel panic (unrelated to ZFS.) The reboot caused the device numbers to shuffle, so I did an 'export/import' to re-read the metadata and get the array back up. Once I swapped drives, I issued a 'zpool replace'. That was 4 days ago now. The progress in a 'zpool status' looks like this, as of right now: scrub: resilver in progress for 0h0m, 0.00% done, 2251h0m to go ... which is a little concerning, since a) it appears to have not moved since I started it, and b) I'm in a DEGRADED state until it finishes... if it finishes. So, I reach out to the list! - Is the resilver progress notification in a known weird state under FreeBSD? - Anything I can do to kick this in the pants? Tuning params? - This was my first drive failure under ZFS -- anything I should have done differently? Such as NOT doing the export/import? (Not sure what else I could have done there.) Some additional info is below. Drives are at about 20% busy, according to vmstat. Seem to have bandwidth to spare. This is a FreeBSD 7.2-STABLE system from the end of May -- 32 bit, 2G of RAM. I have the luxury of this being a test machine (for exactly stuff like this), so I'm willing to try whatever without worrying about production data or SLA. :) -- Mahlon E. Smith http://www.martini.nu/contact.html ----------------------------------------------------------------------- % zfs list store NAME USED AVAIL REFER MOUNTPOINT store 1.22T 2.36T 32.0K none ----------------------------------------------------------------------- % cat /boot/loader.conf vm.kmem_size_max="768M" vm.kmem_size="768M" vfs.zfs.arc_max="256M" ----------------------------------------------------------------------- % zpool status store pool: store state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress for 0h0m, 0.00% done, 2251h0m to go config: NAME STATE READ WRITE CKSUM store DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 da0 ONLINE 0 0 0 274K resilvered da1 ONLINE 0 0 0 282K resilvered replacing DEGRADED 0 0 0 2025342973333799752 UNAVAIL 3 4.11K 0 was /dev/da2 da8 ONLINE 0 0 0 418K resilvered da2 ONLINE 0 0 0 280K resilvered da3 ONLINE 0 0 0 269K resilvered da4 ONLINE 0 0 0 266K resilvered da5 ONLINE 0 0 0 270K resilvered da6 ONLINE 0 0 0 270K resilvered da7 ONLINE 0 0 0 267K resilvered errors: No known data errors ----------------------------------------------------------------------- % zpool iostat -v capacity operations bandwidth pool used avail read write read write ------------------------- ----- ----- ----- ----- ----- ----- store 1.37T 2.72T 49 106 138K 543K raidz1 1.37T 2.72T 49 106 138K 543K da0 - - 15 62 1017K 79.9K da1 - - 15 62 1020K 80.3K replacing - - 0 103 0 88.3K 2025342973333799752 - - 0 0 1.45K 261 da8 - - 0 79 1.45K 98.2K da2 - - 14 62 948K 80.3K da3 - - 13 62 894K 80.0K da4 - - 14 63 942K 80.3K da5 - - 15 62 992K 80.4K da6 - - 15 62 1000K 80.1K da7 - - 15 62 1022K 80.1K ------------------------- ----- ----- ----- ----- ----- ----- -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 155 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20090707/7ac75b29/attachment.pgp
On Tue, Jul 7, 2009 at 12:56 PM, Mahlon E. Smith <mahlon@martini.nu> wrote:> I've got a 9 sata drive raidz1 array, started at version 6, upgraded to > version 13. I had an apparent drive failure, and then at some point, a > kernel panic (unrelated to ZFS.) The reboot caused the device numbers > to shuffle, so I did an 'export/import' to re-read the metadata and get > the array back up. >This is why we've started using glabel(8) to label our drives, and then add the labels to the pool: # zpool create store raidz1 label/disk01 label/disk02 label/disk03 That way, it does matter where the kernel detects the drives or what the physical device node is called, GEOM picks up the label, and ZFS uses the label.> Once I swapped drives, I issued a 'zpool replace'. >See comment at the end: what's the replace command that you used?> > That was 4 days ago now. The progress in a 'zpool status' looks like > this, as of right now: > > scrub: resilver in progress for 0h0m, 0.00% done, 2251h0m to go > > ... which is a little concerning, since a) it appears to have not moved > since I started it, and b) I'm in a DEGRADED state until it finishes... > if it finishes. >There's something wrong here. It definitely should be incrementing. Even when we did the foolish thing of creating a 24-drive raidz2 vdev and had to replace a drive, the progress bar did change. Never got above 39% as it kept restarting, but it did increment.> > So, I reach out to the list! > > - Is the resilver progress notification in a known weird state under > FreeBSD? > > - Anything I can do to kick this in the pants? Tuning params? >I'd redo the replace command, and check the output of "zpool status" to make sure it's showing the proper device node and not some random string of numbers like it is.> - This was my first drive failure under ZFS -- anything I should have > done differently? Such as NOT doing the export/import? (Not sure > what else I could have done there.) >If you knew which drive it was, I'd have shutdown the server and replaced it, so that the drives came back up renumbered correctly. This happened to us once when I was playing around with simulating dead drives (pulling drives) and rebooting. That's when I moved over to using glabels. % zpool status store> pool: store > state: DEGRADED > status: One or more devices is currently being resilvered. The pool will > continue to function, possibly in a degraded state. > action: Wait for the resilver to complete. > scrub: resilver in progress for 0h0m, 0.00% done, 2251h0m to go > config: > > NAME STATE READ WRITE CKSUM > store DEGRADED 0 0 0 > raidz1 DEGRADED 0 0 0 > da0 ONLINE 0 0 0 274K > resilvered > da1 ONLINE 0 0 0 282K > resilvered > replacing DEGRADED 0 0 0 > 2025342973333799752 UNAVAIL 3 4.11K 0 was /dev/da2 > da8 ONLINE 0 0 0 418K > resilvered > da2 ONLINE 0 0 0 280K > resilvered > da3 ONLINE 0 0 0 269K > resilvered > da4 ONLINE 0 0 0 266K > resilvered > da5 ONLINE 0 0 0 270K > resilvered > da6 ONLINE 0 0 0 270K > resilvered > da7 ONLINE 0 0 0 267K > resilvered > > errors: No known data errors > > > ----------------------------------------------------------------------- > > > % zpool iostat -v > capacity operations bandwidth > pool used avail read write read write > ------------------------- ----- ----- ----- ----- ----- ----- > store 1.37T 2.72T 49 106 138K 543K > raidz1 1.37T 2.72T 49 106 138K 543K > da0 - - 15 62 1017K 79.9K > da1 - - 15 62 1020K 80.3K > replacing - - 0 103 0 88.3K > 2025342973333799752 - - 0 0 1.45K 261 > da8 - - 0 79 1.45K 98.2K > da2 - - 14 62 948K 80.3K > da3 - - 13 62 894K 80.0K > da4 - - 14 63 942K 80.3K > da5 - - 15 62 992K 80.4K > da6 - - 15 62 1000K 80.1K > da7 - - 15 62 1022K 80.1K > ------------------------- ----- ----- ----- ----- ----- ----- >That definitely doesn't look right. It should be showing the device name there in the "replacing" section. What's the exact "zpool replace" command that you used? -- Freddie Cash fjwcash@gmail.com
On Tue, Jul 07, 2009 at 12:56:14PM -0700, Mahlon E. Smith wrote:> > I've got a 9 sata drive raidz1 array, started at version 6, upgraded to > version 13. I had an apparent drive failure, and then at some point, a > kernel panic (unrelated to ZFS.) The reboot caused the device numbers > to shuffle, so I did an 'export/import' to re-read the metadata and get > the array back up. > > Once I swapped drives, I issued a 'zpool replace'. > > That was 4 days ago now. The progress in a 'zpool status' looks like > this, as of right now: > > scrub: resilver in progress for 0h0m, 0.00% done, 2251h0m to go > > ... which is a little concerning, since a) it appears to have not moved > since I started it, and b) I'm in a DEGRADED state until it finishes... > if it finishes. > > So, I reach out to the list! > > - Is the resilver progress notification in a known weird state under > FreeBSD? > > - Anything I can do to kick this in the pants? Tuning params? > > - This was my first drive failure under ZFS -- anything I should have > done differently? Such as NOT doing the export/import? (Not sure > what else I could have done there.)I'm seeing essentially the same think on an 8.0-BETA1 box with an 8-disk raidz1 pool. Every once in a while the system makes it to 0.05% done and gives a vaguely reasonable rebuild time, but it quickly drops back to reports 0.00% and it's basically not making any forward progress. In my case this is a copy of a mirror so while it would be a bit annoying to rebuild, the system could be rebuilt fairly easily. On thing I did just notice is that my zpool version is 13, but my file systems are all v1 rather than the latest (v3). I don't know if this is relevant or not. -- Brooks -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20090707/d66e898b/attachment.pgp
> On Tue, Jul 7, 2009 at 3:26 PM, Mahlon E. Smith <mahlon at martini.nu <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>> wrote:*> > On Tue, Jul 07, 2009, Freddie Cash wrote: *> >* > *> >* > This is why we've started using glabel(8) to label our drives, and then *> >* add *> >* > the labels to the pool: *> >* > # zpool create store raidz1 label/disk01 label/disk02 label/disk03 *> >* > *> >* > That way, it does matter where the kernel detects the drives or what the *> >* > physical device node is called, GEOM picks up the label, and ZFS uses the *> >* > label. *> >* *> >* Ah, slick. I'll definitely be doing that moving forward. Wonder if I *> >* could do it piecemeal now via a shell game, labeling and replacing each *> >* individual drive? Will put that on my "try it" list. *> >* *> Yes, this can be done piecemeal, after the fact, on an already configured > pool. That's how I did it on one of our servers. It was originally> configured using the device node names (da0, da1, etc). Then I set up the > second server, but used labels. Then I went back to the first server, > labelled the drives, and did "zpool replace storage da0 label/disk01" for> each drive. Doesn't take long to resilver, as it knows that it's the same > device.It seems a very good practice but how did you do actually on your first already configured server? I am struggling with the followings: 1. on online disk, member of configured raidz pool, "glabel label" fails with error message "operation not permitted". 2. if I make disk offline, glabel succeed, but making disk back to online clears label. 3. if I make disk offline, glabel succeed, but "zpool replace <pool> <dev> <label>" fails with "/dev/label/<label> is part of active pool <pool>" 4. export <pool>, glabel, import <pool> neither works. A detailed guide in "for dummies" style would be appreciated. Thanks, Gabor