Lamp Zy
2011-Apr-25 23:56 UTC
[zfs-discuss] Spare drives sitting idle in raidz2 with failed drive
Hi, One of my drives failed in Raidz2 with two hot spares: # zpool status pool: fwgpool0 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using ''zpool online''. see: http://www.sun.com/msg/ZFS-8000-2Q scrub: resilver completed after 0h0m with 0 errors on Mon Apr 25 14:45:44 2011 config: NAME STATE READ WRITE CKSUM fwgpool0 DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 c4t5000C500108B406Ad0 ONLINE 0 0 0 c4t5000C50010F436E2d0 ONLINE 0 0 0 c4t5000C50011215B6Ed0 ONLINE 0 0 0 c4t5000C50011234715d0 ONLINE 0 0 0 c4t5000C50011252B4Ad0 ONLINE 0 0 0 c4t5000C500112749EDd0 ONLINE 0 0 0 c4t5000C5001128FE4Dd0 UNAVAIL 0 0 0 cannot open c4t5000C500112C4959d0 ONLINE 0 0 0 c4t5000C50011318199d0 ONLINE 0 0 0 c4t5000C500113C0E9Dd0 ONLINE 0 0 0 c4t5000C500113D0229d0 ONLINE 0 0 0 c4t5000C500113E97B8d0 ONLINE 0 0 0 c4t5000C50014D065A9d0 ONLINE 0 0 0 c4t5000C50014D0B3B9d0 ONLINE 0 0 0 c4t5000C50014D55DEFd0 ONLINE 0 0 0 c4t5000C50014D642B7d0 ONLINE 0 0 0 c4t5000C50014D64521d0 ONLINE 0 0 0 c4t5000C50014D69C14d0 ONLINE 0 0 0 c4t5000C50014D6B2CFd0 ONLINE 0 0 0 c4t5000C50014D6C6D7d0 ONLINE 0 0 0 c4t5000C50014D6D486d0 ONLINE 0 0 0 c4t5000C50014D6D77Fd0 ONLINE 0 0 0 spares c4t5000C50014D70072d0 AVAIL c4t5000C50014D7058Dd0 AVAIL errors: No known data errors I''d expect the spare drives to auto-replace the failed one but this is not happening. What am I missing? I really would like to get the pool back in a healthy state using the spare drives before trying to identify which one is the failed drive in the storage array and trying to replace it. How do I do this? Thanks for any hints. -- Peter
Brandon High
2011-Apr-26 00:47 UTC
[zfs-discuss] Spare drives sitting idle in raidz2 with failed drive
On Mon, Apr 25, 2011 at 4:56 PM, Lamp Zy <lampzy at gmail.com> wrote:> I''d expect the spare drives to auto-replace the failed one but this is not > happening. > > What am I missing?Is the autoreplace property set to ''on''? # zpool get autoreplace fwgpool0 # zpool set autoreplace=on fwgpool0> I really would like to get the pool back in a healthy state using the spare > drives before trying to identify which one is the failed drive in the > storage array and trying to replace it. How do I do this?Turning on autoreplace might start the replace. If not, the following will replace the failed drive with the first spare. (I''d suggest verifying the device names before running it.) # zpool replace fwgpool0 c4t5000C5001128FE4Dd0 c4t5000C50014D70072d0 -B -- Brandon High : bhigh at freaks.com
Lamp Zy
2011-Apr-26 02:01 UTC
[zfs-discuss] Spare drives sitting idle in raidz2 with failed drive
Thanks Brandon, On 04/25/2011 05:47 PM, Brandon High wrote:> On Mon, Apr 25, 2011 at 4:56 PM, Lamp Zy<lampzy at gmail.com> wrote: >> I''d expect the spare drives to auto-replace the failed one but this is not >> happening. >> >> What am I missing? > > Is the autoreplace property set to ''on''? > # zpool get autoreplace fwgpool0 > # zpool set autoreplace=on fwgpool0Yes, autoreplace is on. I should have mentioned it in my original post: # zpool get autoreplace fwgpool0 NAME PROPERTY VALUE SOURCE fwgpool0 autoreplace on local>> I really would like to get the pool back in a healthy state using the spare >> drives before trying to identify which one is the failed drive in the >> storage array and trying to replace it. How do I do this? > > Turning on autoreplace might start the replace. If not, the following > will replace the failed drive with the first spare. (I''d suggest > verifying the device names before running it.) > # zpool replace fwgpool0 c4t5000C5001128FE4Dd0 c4t5000C50014D70072d0I thought about doing that. My understanding is that this command should be used to replace a drive with a brand new one i.e. a drive that is not known to the raidz configuration. Should I somehow unconfigure one of the spare drives to be just a loose drive and not a raidz spare before running the command (and how do I do it)? Or, is it save to just run the replace command and let zfs take care of the details like noticing that one of the spares has been manually re-purposed to replace a failed drive? Thank you Peter
Nikola M.
2011-Apr-26 08:25 UTC
[zfs-discuss] Spare drives sitting idle in raidz2 with failed drive
On 04/26/11 01:56 AM, Lamp Zy wrote:> Hi, > > One of my drives failed in Raidz2 with two hot spares:What are zpool/zfs versions? (zpool upgrade Ctrl+c, zfs upgrade Cttr+c). Latest zpool/zfs versions available by numerical designation in all OpenSolaris based distributions, are zpool 28 and zfs v. 5. (That is why one should Not update so S11Ex Zfs/Zpool version if wanting to use/have installed or continue using in multiple Zfs BE''s other open OpenSolaris based distributions) What OS are you using with ZFS? Do you use Solaris 10/update release, Solaris11Express, OpenIndiana oi_148 dev/ 148b with IllumOS, OpenSolaris 2009.06/snv_134b, Nexenta, Nexenta Community, Schillix, FreeBSD, Linux zfs-fuse.. (I guess still not using Linux with Zfs kernel module, but just to mention it available.. and OSX too).
Cindy Swearingen
2011-Apr-26 15:22 UTC
[zfs-discuss] Spare drives sitting idle in raidz2 with failed drive
Hi-- I don''t know why the spare isn''t kicking in automatically, it should. A documented workaround is to outright replace the failed disk with one of the spares, like this: # zpool replace fwgpool0 c4t5000C5001128FE4Dd0 c4t5000C50014D70072d0 The autoreplace pool property has nothing to do with automatic spare replacement. When this property is enabled, a replacement disk will be automatically labeled and replaced. No need to manually run the zpool command when this property is enabled. Then, you can find the original failed c4t5000C5001128FE4Dd0 disk and physically replace it when you have time. You could then add this disk back into the pool as the new spare, like this: # zpool add fwgpool0 spare c4t5000C5001128FE4Dd0 Thanks, Cindy On 04/25/11 17:56, Lamp Zy wrote:> Hi, > > One of my drives failed in Raidz2 with two hot spares: > > # zpool status > pool: fwgpool0 > state: DEGRADED > status: One or more devices could not be opened. Sufficient replicas > exist for > the pool to continue functioning in a degraded state. > action: Attach the missing device and online it using ''zpool online''. > see: http://www.sun.com/msg/ZFS-8000-2Q > scrub: resilver completed after 0h0m with 0 errors on Mon Apr 25 > 14:45:44 2011 > config: > > NAME STATE READ WRITE CKSUM > fwgpool0 DEGRADED 0 0 0 > raidz2 DEGRADED 0 0 0 > c4t5000C500108B406Ad0 ONLINE 0 0 0 > c4t5000C50010F436E2d0 ONLINE 0 0 0 > c4t5000C50011215B6Ed0 ONLINE 0 0 0 > c4t5000C50011234715d0 ONLINE 0 0 0 > c4t5000C50011252B4Ad0 ONLINE 0 0 0 > c4t5000C500112749EDd0 ONLINE 0 0 0 > c4t5000C5001128FE4Dd0 UNAVAIL 0 0 0 cannot open > c4t5000C500112C4959d0 ONLINE 0 0 0 > c4t5000C50011318199d0 ONLINE 0 0 0 > c4t5000C500113C0E9Dd0 ONLINE 0 0 0 > c4t5000C500113D0229d0 ONLINE 0 0 0 > c4t5000C500113E97B8d0 ONLINE 0 0 0 > c4t5000C50014D065A9d0 ONLINE 0 0 0 > c4t5000C50014D0B3B9d0 ONLINE 0 0 0 > c4t5000C50014D55DEFd0 ONLINE 0 0 0 > c4t5000C50014D642B7d0 ONLINE 0 0 0 > c4t5000C50014D64521d0 ONLINE 0 0 0 > c4t5000C50014D69C14d0 ONLINE 0 0 0 > c4t5000C50014D6B2CFd0 ONLINE 0 0 0 > c4t5000C50014D6C6D7d0 ONLINE 0 0 0 > c4t5000C50014D6D486d0 ONLINE 0 0 0 > c4t5000C50014D6D77Fd0 ONLINE 0 0 0 > spares > c4t5000C50014D70072d0 AVAIL > c4t5000C50014D7058Dd0 AVAIL > > errors: No known data errors > > > I''d expect the spare drives to auto-replace the failed one but this is > not happening. > > What am I missing? > > I really would like to get the pool back in a healthy state using the > spare drives before trying to identify which one is the failed drive in > the storage array and trying to replace it. How do I do this? > > Thanks for any hints. > > -- > Peter > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Richard Elling
2011-Apr-26 20:59 UTC
[zfs-discuss] Spare drives sitting idle in raidz2 with failed drive
On Apr 26, 2011, at 8:22 AM, Cindy Swearingen wrote:> Hi-- > > I don''t know why the spare isn''t kicking in automatically, it should.This can happen if the FMA agents aren''t working properly. FYI, in NexentaStor we have added a zfs-monitor FMA agent to check the health of disks in use for ZFS and notice when they are no longer responding to reads. -- richard
Paul Kraus
2011-Apr-26 21:47 UTC
[zfs-discuss] Spare drives sitting idle in raidz2 with failed drive
On Tue, Apr 26, 2011 at 4:59 PM, Richard Elling <richard.elling at gmail.com> wrote:> > On Apr 26, 2011, at 8:22 AM, Cindy Swearingen wrote: > >> Hi-- >> >> I don''t know why the spare isn''t kicking in automatically, it should. > > This can happen if the FMA agents aren''t working properly. > > FYI, in NexentaStor we have added a zfs-monitor FMA agent to check the > health of disks in use for ZFS and notice when they are no longer responding > to reads.I just recently (this past week) had a very similar failure. zpool consisting of two raidz2 vdevs and two hot spare drives. Each raidz2 vdev consists of 10 drives (I know, not the best layout, but the activity is large sequential writes and reads and we needed the capacity). We had a drive fail in one of the vdevs and one of the hot spares automatically went into action (the special spare device within the vdev came into being and the hot spare drive resilvered). A short time later a second drive in the same vdev failed. No action by any hot spare. The system was running Solaris 10U8 with no additional patches. I opened a case with Oracle and they told me that the hot spare *should* have dealt with the second failure. We replaced the first (hot spared) drive with zpool replace and it resilvered fine. Then we replaced the second (non hot spared) drive with zpool replace and the system hung. I suspected the mpt (multipathing) driver for the SATA drives in the J4400, there have been some huge improvements in that driver since 10U8. After rebooting the drive appeared replaced and was resilvering. Oracle support chocked the hot spare issue up to an FMA problem but could not duplicate it in the lab. We have since upgraded to 10U9 + the latest CPU (April 2011) and are hoping both the hot spare issue and the mpt drive issue are fixed. -- {--------1---------2---------3---------4---------5---------6---------7---------} Paul Kraus -> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ ) -> Sound Coordinator, Schenectady Light Opera Company ( http://www.sloctheater.org/ ) -> Technical Advisor, RPI Players
Edward Ned Harvey
2011-Apr-27 14:32 UTC
[zfs-discuss] Spare drives sitting idle in raidz2 with failed drive
> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss- > bounces at opensolaris.org] On Behalf Of Lamp Zy > > One of my drives failed in Raidz2 with two hot spares: >What zpool & zfs version are you using? What OS version? Are all the drives precisely the same size (Same make/model number?) and all the same firmware level? Up to some point (I don''t know which zpool version) there was a characteristic (some would say a bug) whereby even a byte smaller drive would cause the new drive to be an unsuitable replacement for a failed drive. And it was certainly known to happen sometimes, that a single mfgr & model of drive would occasionally have these tiny variations in supposedly identical drives. But they created a workaround for this in some version of zpool.
Lamp Zy
2011-Apr-27 19:51 UTC
[zfs-discuss] Spare drives sitting idle in raidz2 with failed drive
On 04/26/2011 01:25 AM, Nikola M. wrote:> On 04/26/11 01:56 AM, Lamp Zy wrote: >> Hi, >> >> One of my drives failed in Raidz2 with two hot spares: > What are zpool/zfs versions? (zpool upgrade Ctrl+c, zfs upgrade Cttr+c). > Latest zpool/zfs versions available by numerical designation in all > OpenSolaris based distributions, are zpool 28 and zfs v. 5. (That is why > one should Not update so S11Ex Zfs/Zpool version if wanting to use/have > installed or continue using in multiple Zfs BE''s other open OpenSolaris > based distributions) > > What OS are you using with ZFS? > Do you use Solaris 10/update release, Solaris11Express, OpenIndiana > oi_148 dev/ 148b with IllumOS, OpenSolaris 2009.06/snv_134b, Nexenta, > Nexenta Community, Schillix, FreeBSD, Linux zfs-fuse.. (I guess still > not using Linux with Zfs kernel module, but just to mention it > available.. and OSX too).Thank you for all replies. Here is what we are using. - Hardware: Server: SUN SunFire X4240 DAS Storage: SUN Storage J4400 with 24x1TB SATA drives. Original drives. I assume they are identical. - Software: OS: Solaris 10 5/09 s10x_u7wos_08 X86; Stock install. No upgrades, no patches. ZFS pool version 10 ZFS filesystem version 3 Another confusing thing is that I wasn''t able to put the failed drive off-line because there wasn''t enough replicas (?). First, the drive already failed and second - it''s raidz2 which is equivalent of raid6 and it should be able to handle 2 failed drives. I skipped that step but wanted to mention it here. I used the "zpool replace" and resilvering finished successfully. Then the "zpool detach" removed the drive and now I have this: # zpool status fwgpool0 pool: fwgpool0 state: ONLINE scrub: resilver completed after 12h59m with 0 errors on Wed Apr 27 05:15:17 2011 config: NAME STATE READ WRITE CKSUM fwgpool0 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 c4t5000C500108B406Ad0 ONLINE 0 0 0 c4t5000C50010F436E2d0 ONLINE 0 0 0 c4t5000C50011215B6Ed0 ONLINE 0 0 0 c4t5000C50011234715d0 ONLINE 0 0 0 c4t5000C50011252B4Ad0 ONLINE 0 0 0 c4t5000C500112749EDd0 ONLINE 0 0 0 c4t5000C50014D70072d0 ONLINE 0 0 0 c4t5000C500112C4959d0 ONLINE 0 0 0 c4t5000C50011318199d0 ONLINE 0 0 0 c4t5000C500113C0E9Dd0 ONLINE 0 0 0 c4t5000C500113D0229d0 ONLINE 0 0 0 c4t5000C500113E97B8d0 ONLINE 0 0 0 c4t5000C50014D065A9d0 ONLINE 0 0 0 c4t5000C50014D0B3B9d0 ONLINE 0 0 0 c4t5000C50014D55DEFd0 ONLINE 0 0 0 c4t5000C50014D642B7d0 ONLINE 0 0 0 c4t5000C50014D64521d0 ONLINE 0 0 0 c4t5000C50014D69C14d0 ONLINE 0 0 0 c4t5000C50014D6B2CFd0 ONLINE 0 0 0 c4t5000C50014D6C6D7d0 ONLINE 0 0 0 c4t5000C50014D6D486d0 ONLINE 0 0 0 c4t5000C50014D6D77Fd0 ONLINE 0 0 0 spares c4t5000C50014D7058Dd0 AVAIL errors: No known data errors # Great. So, now how do I identify which drive out of the 24 in the storage unit is the one that failed? I looked on the Internet for help but the problem is that this drive completely disappeared. Even "format" and "iostat -En" show only 23 drives when there are physically 24. Any ideas how to identify which drive is the one that failed so I can replace it? Thanks Peter
Brandon High
2011-Apr-27 19:59 UTC
[zfs-discuss] Spare drives sitting idle in raidz2 with failed drive
On Wed, Apr 27, 2011 at 12:51 PM, Lamp Zy <lampzy at gmail.com> wrote:> Any ideas how to identify which drive is the one that failed so I can > replace it?Try the following: # fmdump -eV # fmadm faulty -B -- Brandon High : bhigh at freaks.com
Paul Kraus
2011-Apr-27 20:01 UTC
[zfs-discuss] Spare drives sitting idle in raidz2 with failed drive
On Wed, Apr 27, 2011 at 3:51 PM, Lamp Zy <lampzy at gmail.com> wrote:> Great. So, now how do I identify which drive out of the 24 in the storage > unit is the one that failed? > > I looked on the Internet for help but the problem is that this drive > completely disappeared. Even "format" and "iostat -En" show only 23 drives > when there are physically 24. > > Any ideas how to identify which drive is the one that failed so I can > replace it?We are using CAM to monitor our J4400s and through that interface you can see which drive is in which slot. http://www.oracle.com/us/products/servers-storage/storage/storage-software/031603.htm -- {--------1---------2---------3---------4---------5---------6---------7---------} Paul Kraus -> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ ) -> Sound Coordinator, Schenectady Light Opera Company ( http://www.sloctheater.org/ ) -> Technical Advisor, RPI Players