Looks like I''ve hit this bug:
http://bugs.opensolaris.org/view_bug.do?bug_id=6782540 However, none of the
workaround listed in that bug, or any of the related bugs, works. :(
Going through the zfs-discuss and freebsd-fs archives, I see that others
have run into this issue, and managed to solve it via "zpool detach/zpool
online/zpool replace". However, looking closely at the archived messages,
all the successful tests had one thing in common: 1 drive ONLINE, 1 drive
FAULTED. If a drive is online, obviously it can be detached. In all the
cases where people have been unsuccessful at fixing this situation, 1 drive
is OFFLINE, and 1 drive is FAULTED. As is our case:
[fcash at thehive ~]$ zpool status -v
pool: storage
state: DEGRADED
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using ''zpool upgrade''. Once this is
done, the
pool will no longer be accessible on older software versions.
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
storage DEGRADED 0 0 0
raidz2 DEGRADED 0 0 0
label/disk01 ONLINE 0 0 0
label/disk02 ONLINE 0 0 0
label/disk03 ONLINE 0 0 0
replacing UNAVAIL 0 534 0 insufficient
replicas
label/disk04/old OFFLINE 0 544 0
label/disk04 FAULTED 0 544 0 corrupted data
label/disk13 ONLINE 0 0 0
label/disk14 ONLINE 0 0 0
label/disk15 ONLINE 0 0 0
label/disk16 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
label/disk05 ONLINE 0 0 0
label/disk06 ONLINE 0 0 0
label/disk07 ONLINE 0 0 0
label/disk08 ONLINE 0 0 0
label/disk17 ONLINE 0 0 0
label/disk18 ONLINE 0 0 0
label/disk19 ONLINE 0 0 0
label/disk20 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
label/disk09 ONLINE 0 0 0
label/disk10 ONLINE 0 0 0
label/disk11 ONLINE 0 0 0
label/disk12 ONLINE 0 0 0
label/disk21 ONLINE 0 0 0
label/disk22 ONLINE 0 0 0
label/disk23 ONLINE 0 0 0
label/disk24 ONLINE 0 0 0
cache
label/cache ONLINE 0 0 0
errors: No known data errors
[fcash at thehive ~]$ sudo zpool replace storage label/disk04
cannot replace label/disk04 with label/disk04: cannot replace a replacing
device
Note the OFFLINE status for label/disk04/old. I cannot get either drive to
detach, or to replace, or to online, or to offline. "zpool online" on
the
old device changes the status to UNAVAIL. "zpool detach" and
"zpool
offline" give the same error: no valid replicas.
I''ve tried removing the underlying device, booting with the drive in
the
system and without the drive in the system, all kinds of zpool commands, all
without success.
Is there any way to recover from this error? Or am I doomed to destroy a 10
TB pool?
FreeBSD thehive.sd73.bc.ca 8.0-STABLE FreeBSD 8.0-STABLE #3: Fri Jan 15
11:08:47 PST 2010 root at thehive.sd73.bc.ca:/usr/obj/usr/src-8/sys/ZFSHOST
amd64
ZFSv13 (ZFSv14 is available)
--
Freddie Cash
fjwcash at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100428/16521e86/attachment.html>
Victor Latushkin
2010-Apr-28 21:48 UTC
[zfs-discuss] ZFS: "Cannot replace a replacing drive"
On Apr 28, 2010, at 8:00 PM, Freddie Cash wrote:> Looks like I''ve hit this bug: http://bugs.opensolaris.org/view_bug.do?bug_id=6782540 However, none of the workaround listed in that bug, or any of the related bugs, works. :( > > Going through the zfs-discuss and freebsd-fs archives, I see that others have run into this issue, and managed to solve it via "zpool detach/zpool online/zpool replace". However, looking closely at the archived messages, all the successful tests had one thing in common: 1 drive ONLINE, 1 drive FAULTED. If a drive is online, obviously it can be detached. In all the cases where people have been unsuccessful at fixing this situation, 1 drive is OFFLINE, and 1 drive is FAULTED. As is our case: > > [fcash at thehive ~]$ zpool status -v > pool: storage > state: DEGRADED > status: The pool is formatted using an older on-disk format. The pool can > still be used, but some features are unavailable. > action: Upgrade the pool using ''zpool upgrade''. Once this is done, the > pool will no longer be accessible on older software versions. > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > storage DEGRADED 0 0 0 > raidz2 DEGRADED 0 0 0 > label/disk01 ONLINE 0 0 0 > label/disk02 ONLINE 0 0 0 > label/disk03 ONLINE 0 0 0 > replacing UNAVAIL 0 534 0 insufficient replicas > label/disk04/old OFFLINE 0 544 0 > label/disk04 FAULTED 0 544 0 corrupted data > label/disk13 ONLINE 0 0 0 > label/disk14 ONLINE 0 0 0 > label/disk15 ONLINE 0 0 0 > label/disk16 ONLINE 0 0 0 > raidz2 ONLINE 0 0 0 > label/disk05 ONLINE 0 0 0 > label/disk06 ONLINE 0 0 0 > label/disk07 ONLINE 0 0 0 > label/disk08 ONLINE 0 0 0 > label/disk17 ONLINE 0 0 0 > label/disk18 ONLINE 0 0 0 > label/disk19 ONLINE 0 0 0 > label/disk20 ONLINE 0 0 0 > raidz2 ONLINE 0 0 0 > label/disk09 ONLINE 0 0 0 > label/disk10 ONLINE 0 0 0 > label/disk11 ONLINE 0 0 0 > label/disk12 ONLINE 0 0 0 > label/disk21 ONLINE 0 0 0 > label/disk22 ONLINE 0 0 0 > label/disk23 ONLINE 0 0 0 > label/disk24 ONLINE 0 0 0 > cache > label/cache ONLINE 0 0 0 > > errors: No known data errors > > [fcash at thehive ~]$ sudo zpool replace storage label/disk04 > cannot replace label/disk04 with label/disk04: cannot replace a replacing device > > > Note the OFFLINE status for label/disk04/old. I cannot get either drive to detach, or to replace, or to online, or to offline. "zpool online" on the old device changes the status to UNAVAIL. "zpool detach" and "zpool offline" give the same error: no valid replicas. > > I''ve tried removing the underlying device, booting with the drive in the system and without the drive in the system, all kinds of zpool commands, all without success. > > Is there any way to recover from this error? Or am I doomed to destroy a 10 TB pool? > > FreeBSD thehive.sd73.bc.ca 8.0-STABLE FreeBSD 8.0-STABLE #3: Fri Jan 15 11:08:47 PST 2010 root at thehive.sd73.bc.ca:/usr/obj/usr/src-8/sys/ZFSHOST amd64 > > ZFSv13 (ZFSv14 is available)Can you provide the following information: 1. Output of zdb -l /path/to/disk for FAULTED disk 2. Run ''zdb -ddd storage'' and provide section titles Dirty Time Logs 3. Try ''zpool detach'' approach on v14 system? Another option may be to try latest OpenSolaris LiveCD (build 134). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100429/01a7ac12/attachment.html>
On Wed, Apr 28, 2010 at 2:48 PM, Victor Latushkin <Victor.Latushkin at sun.com>wrote:> Can you provide the following information: > > 1. Output of zdb -l /path/to/disk for FAULTED disk > > [root at thehive ~]# zdb -l /dev/label/disk04-------------------------------------------- LABEL 0 -------------------------------------------- version=13 name=''storage'' state=0 txg=0 pool_guid=7019877621158930327 hostid=2180312168 hostname=''thehive.sd73.bc.ca'' top_guid=1237079380739157224 guid=1237079380739157224 vdev_tree type=''disk'' id=0 guid=1237079380739157224 path=''/dev/label/disk04'' whole_disk=0 metaslab_array=0 metaslab_shift=0 ashift=9 asize=1499984035840 is_log=0 create_txg=5125455 -------------------------------------------- LABEL 1 -------------------------------------------- version=13 name=''storage'' state=0 txg=0 pool_guid=7019877621158930327 hostid=2180312168 hostname=''thehive.sd73.bc.ca'' top_guid=1237079380739157224 guid=1237079380739157224 vdev_tree type=''disk'' id=0 guid=1237079380739157224 path=''/dev/label/disk04'' whole_disk=0 metaslab_array=0 metaslab_shift=0 ashift=9 asize=1499984035840 is_log=0 create_txg=5125455 -------------------------------------------- LABEL 2 -------------------------------------------- version=13 name=''storage'' state=0 txg=0 pool_guid=7019877621158930327 hostid=2180312168 hostname=''thehive.sd73.bc.ca'' top_guid=1237079380739157224 guid=1237079380739157224 vdev_tree type=''disk'' id=0 guid=1237079380739157224 path=''/dev/label/disk04'' whole_disk=0 metaslab_array=0 metaslab_shift=0 ashift=9 asize=1499984035840 is_log=0 create_txg=5125455 -------------------------------------------- LABEL 3 -------------------------------------------- version=13 name=''storage'' state=0 txg=0 pool_guid=7019877621158930327 hostid=2180312168 hostname=''thehive.sd73.bc.ca'' top_guid=1237079380739157224 guid=1237079380739157224 vdev_tree type=''disk'' id=0 guid=1237079380739157224 path=''/dev/label/disk04'' whole_disk=0 metaslab_array=0 metaslab_shift=0 ashift=9 asize=1499984035840 is_log=0 create_txg=5125455> 2. Run ''zdb -ddd storage'' and provide section titles Dirty Time Logs > > See attached.3. Try ''zpool detach'' approach on v14 system?> > Pool upgraded successfully.Same results to all the zpool commands, though: online, offline, detach, replace.> Another option may be to try latest OpenSolaris LiveCD (build 134). > > I''ll have to see if I can download/make one.Does it include drivers for 3Ware 9550SXU and 9650SE RAID controllers? All the drives are plugged into those. -- Freddie Cash fjwcash at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100428/2b06ce17/attachment.html> -------------- next part -------------- Dirty time logs: storage outage [5120292,5124552] length 4261 outage [5124554,5124554] length 1 outage [5124556,5124558] length 3 outage [5124560,5124562] length 3 outage [5124564,5124564] length 1 outage [5124566,5124568] length 3 outage [5124570,5124572] length 3 outage [5124574,5124574] length 1 outage [5124576,5124582] length 7 outage [5124584,5124584] length 1 outage [5124586,5124588] length 3 outage [5124590,5124592] length 3 outage [5124594,5124594] length 1 outage [5124596,5124598] length 3 outage [5124600,5124602] length 3 outage [5124604,5124604] length 1 outage [5124606,5124608] length 3 outage [5124610,5124612] length 3 outage [5124614,5124614] length 1 outage [5124616,5124618] length 3 outage [5124620,5124624] length 5 outage [5124626,5124628] length 3 outage [5124630,5124632] length 3 outage [5124634,5124634] length 1 outage [5124636,5124638] length 3 outage [5124640,5124642] length 3 outage [5124644,5124648] length 5 outage [5124650,5124652] length 3 outage [5124654,5124654] length 1 outage [5124656,5124658] length 3 outage [5124660,5124662] length 3 outage [5124664,5124664] length 1 outage [5124666,5124668] length 3 outage [5124670,5124672] length 3 outage [5124674,5124674] length 1 outage [5124676,5124678] length 3 outage [5124680,5124682] length 3 outage [5124684,5124684] length 1 outage [5124686,5124688] length 3 outage [5124690,5124692] length 3 outage [5124694,5124694] length 1 outage [5124696,5124710] length 15 outage [5124712,5124712] length 1 outage [5124714,5124721] length 8 outage [5124723,5124723] length 1 outage [5124725,5124727] length 3 outage [5124729,5124729] length 1 outage [5124731,5124731] length 1 outage [5124733,5124737] length 5 outage [5124739,5124739] length 1 outage [5124741,5124747] length 7 outage [5124749,5124749] length 1 outage [5124751,5124751] length 1 outage [5124753,5124753] length 1 outage [5124755,5124757] length 3 outage [5124759,5124768] length 10 outage [5124770,5124770] length 1 outage [5124772,5124772] length 1 outage [5124774,5124774] length 1 outage [5124776,5124784] length 9 outage [5124786,5124791] length 6 outage [5124793,5124793] length 1 outage [5124795,5124795] length 1 outage [5124797,5124797] length 1 outage [5124799,5124799] length 1 outage [5124801,5124801] length 1 outage [5124803,5124803] length 1 outage [5124805,5124805] length 1 outage [5124807,5124807] length 1 outage [5124809,5124809] length 1 outage [5124811,5124811] length 1 outage [5124813,5124813] length 1 outage [5124815,5124815] length 1 outage [5124817,5124817] length 1 outage [5124819,5124819] length 1 outage [5124821,5124821] length 1 outage [5124823,5124823] length 1 outage [5124825,5124825] length 1 outage [5124827,5124827] length 1 outage [5124829,5124829] length 1 outage [5124831,5124831] length 1 outage [5124833,5124833] length 1 outage [5124835,5124835] length 1 outage [5124837,5124837] length 1 outage [5124839,5124839] length 1 outage [5124841,5124841] length 1 outage [5124843,5124845] length 3 outage [5124847,5124847] length 1 outage [5124849,5124849] length 1 outage [5124851,5124851] length 1 outage [5124853,5124855] length 3 outage [5124857,5124857] length 1 outage [5124859,5124859] length 1 outage [5124861,5124861] length 1 outage [5124863,5124865] length 3 outage [5124867,5124867] length 1 outage [5124869,5124869] length 1 outage [5124871,5124871] length 1 outage [5124873,5124875] length 3 outage [5124877,5124877] length 1 outage [5124879,5124879] length 1 outage [5124881,5124881] length 1 outage [5124883,5124885] length 3 outage [5124887,5124887] length 1 outage [5124889,5124889] length 1 outage [5124891,5124891] length 1 outage [5124893,5124895] length 3 outage [5124897,5124897] length 1 outage [5124899,5124899] length 1 outage [5124901,5124901] length 1 outage [5124903,5124905] length 3 outage [5124907,5124907] length 1 outage [5124909,5124911] length 3 outage [5124913,5124915] length 3 outage [5124917,5124917] length 1 outage [5124919,5124919] length 1 outage [5124921,5124921] length 1 outage [5124923,5124925] length 3 outage [5124927,5124927] length 1 outage [5124929,5124929] length 1 outage [5124931,5124931] length 1 outage [5124933,5124935] length 3 outage [5124937,5124937] length 1 outage [5124939,5124939] length 1 outage [5124941,5124941] length 1 outage [5124943,5124945] length 3 outage [5124947,5124947] length 1 outage [5124949,5124949] length 1 outage [5124951,5124951] length 1 outage [5124953,5124955] length 3 outage [5124957,5124957] length 1 outage [5124959,5124959] length 1 outage [5124961,5124961] length 1 outage [5124963,5124965] length 3 outage [5124967,5124967] length 1 outage [5124969,5124969] length 1 outage [5124971,5124971] length 1 outage [5124973,5124975] length 3 outage [5124977,5124977] length 1 outage [5124979,5124979] length 1 outage [5124981,5124981] length 1 outage [5124983,5124985] length 3 outage [5124987,5124987] length 1 outage [5124989,5124989] length 1 outage [5124991,5124991] length 1 outage [5124993,5124995] length 3 outage [5124997,5124997] length 1 outage [5124999,5124999] length 1 outage [5125001,5125001] length 1 outage [5125003,5125005] length 3 outage [5125007,5125007] length 1 outage [5125009,5125009] length 1 outage [5125011,5125011] length 1 outage [5125013,5125015] length 3 outage [5125017,5125017] length 1 outage [5125019,5125019] length 1 outage [5125021,5125021] length 1 outage [5125023,5125025] length 3 outage [5125027,5125027] length 1 outage [5125029,5125029] length 1 outage [5125031,5125031] length 1 outage [5125033,5125035] length 3 outage [5125037,5125037] length 1 outage [5125039,5125039] length 1 outage [5125041,5125041] length 1 outage [5125043,5125045] length 3 outage [5125047,5125047] length 1 outage [5125049,5125049] length 1 outage [5125051,5125051] length 1 outage [5125053,5125055] length 3 outage [5125057,5125057] length 1 outage [5125059,5125059] length 1 outage [5125061,5125061] length 1 outage [5125063,5125065] length 3 outage [5125067,5125067] length 1 outage [5125069,5125069] length 1 outage [5125071,5125071] length 1 outage [5125073,5125075] length 3 outage [5125077,5125077] length 1 outage [5125079,5125079] length 1 outage [5125081,5125081] length 1 outage [5125083,5125085] length 3 outage [5125087,5125087] length 1 outage [5125089,5125089] length 1 outage [5125091,5125091] length 1 outage [5125093,5125095] length 3 outage [5125097,5125097] length 1 outage [5125099,5125099] length 1 outage [5125101,5125101] length 1 outage [5125103,5125105] length 3 outage [5125107,5125107] length 1 outage [5125109,5125109] length 1 outage [5125111,5125111] length 1 outage [5125113,5125115] length 3 outage [5125117,5125117] length 1 outage [5125119,5125119] length 1 outage [5125121,5125121] length 1 outage [5125123,5125125] length 3 outage [5125127,5125127] length 1 outage [5125129,5125129] length 1 outage [5125131,5125131] length 1 outage [5125133,5125135] length 3 outage [5125137,5125137] length 1 outage [5125139,5125139] length 1 outage [5125141,5125141] length 1 outage [5125143,5125145] length 3 outage [5125147,5125147] length 1 outage [5125149,5125149] length 1 outage [5125151,5125151] length 1 outage [5125153,5125154] length 2 outage [5125156,5125156] length 1 outage [5125158,5125160] length 3 outage [5125162,5125162] length 1 outage [5125164,5125164] length 1 outage [5125166,5125166] length 1 outage [5125168,5125170] length 3 outage [5125172,5125172] length 1 outage [5125174,5125175] length 2 outage [5125177,5125177] length 1 outage [5125179,5125179] length 1 outage [5125181,5125181] length 1 outage [5125183,5125185] length 3 outage [5125187,5125187] length 1 outage [5125189,5125189] length 1 outage [5125191,5125191] length 1 outage [5125193,5125195] length 3 outage [5125197,5125197] length 1 outage [5125199,5125199] length 1 outage [5125201,5125201] length 1 outage [5125203,5125205] length 3 outage [5125207,5125207] length 1 outage [5125209,5125209] length 1 outage [5125211,5125211] length 1 outage [5125213,5125215] length 3 outage [5125217,5125217] length 1 outage [5125219,5125219] length 1 outage [5125221,5125221] length 1 outage [5125223,5125225] length 3 outage [5125227,5125227] length 1 outage [5125229,5125229] length 1 outage [5125231,5125231] length 1 outage [5125233,5125235] length 3 outage [5125237,5125237] length 1 outage [5125239,5125239] length 1 outage [5125241,5125241] length 1 outage [5125243,5125245] length 3 outage [5125247,5125249] length 3 outage [5125251,5125251] length 1 outage [5125253,5125255] length 3 outage [5125257,5125257] length 1 outage [5125259,5125259] length 1 outage [5125261,5125261] length 1 outage [5125263,5125265] length 3 outage [5125267,5125267] length 1 outage [5125269,5125269] length 1 outage [5125271,5125271] length 1 outage [5125273,5125275] length 3 outage [5125277,5125277] length 1 outage [5125279,5125279] length 1 outage [5125281,5125281] length 1 outage [5125283,5125285] length 3 outage [5125287,5125287] length 1 outage [5125289,5125289] length 1 outage [5125291,5125291] length 1 outage [5125293,5125295] length 3 outage [5125297,5125297] length 1 outage [5125299,5125299] length 1 outage [5125301,5125301] length 1 outage [5125303,5125305] length 3 outage [5125307,5125307] length 1 outage [5125309,5125309] length 1 outage [5125311,5125311] length 1 outage [5125313,5125315] length 3 outage [5125317,5125317] length 1 outage [5125319,5125319] length 1 outage [5125321,5125321] length 1 outage [5125323,5125325] length 3 outage [5125327,5125327] length 1 outage [5125329,5125329] length 1 outage [5125331,5125331] length 1 outage [5125333,5125335] length 3 outage [5125337,5125337] length 1 outage [5125339,5125339] length 1 outage [5125341,5125341] length 1 outage [5125343,5125345] length 3 outage [5125347,5125347] length 1 outage [5125349,5125349] length 1 outage [5125351,5125351] length 1 outage [5125353,5125355] length 3 outage [5125357,5125357] length 1 outage [5125359,5125359] length 1 outage [5125361,5125361] length 1 outage [5125363,5125365] length 3 outage [5125367,5125367] length 1 outage [5125369,5125369] length 1 outage [5125371,5125371] length 1 outage [5125373,5125375] length 3 outage [5125377,5125377] length 1 outage [5125379,5125379] length 1 outage [5125381,5125381] length 1 outage [5125383,5125385] length 3 outage [5125387,5125387] length 1 outage [5125389,5125389] length 1 outage [5125391,5125391] length 1 outage [5125393,5125395] length 3 outage [5125397,5125397] length 1 outage [5125399,5125399] length 1 outage [5125401,5125401] length 1 outage [5125403,5125405] length 3 outage [5125407,5125407] length 1 outage [5125409,5125409] length 1 outage [5125411,5125411] length 1 outage [5125413,5125415] length 3 outage [5125417,5125417] length 1 outage [5125419,5125419] length 1 outage [5125421,5125421] length 1 outage [5125423,5125440] length 18 outage [5125442,5125444] length 3 outage [5125446,5125446] length 1 outage [5125448,5125450] length 3 outage [5125452,5125452] length 1 outage [5125455,5125456] length 2 outage [5125466,5125468] length 3 outage [5125470,5125472] length 3 outage [5125474,5125474] length 1 outage [5125476,5125479] length 4 outage [5125481,5125483] length 3 outage [5125485,5125485] length 1 outage [5125487,5125496] length 10 outage [5125498,5125498] length 1 outage [5125500,5125500] length 1 outage [5125502,5125502] length 1 outage [5125504,5125504] length 1 outage [5125506,5125506] length 1 outage [5125508,5125508] length 1 outage [5125510,5125510] length 1 outage [5125512,5125512] length 1 outage [5125514,5125514] length 1 outage [5125516,5125516] length 1 outage [5125518,5125518] length 1 outage [5125520,5125520] length 1 outage [5125522,5125522] length 1 outage [5125524,5125524] length 1 outage [5125526,5125526] length 1 outage [5125528,5125528] length 1 outage [5125530,5125530] length 1 outage [5125532,5125532] length 1 outage [5125534,5125537] length 4 raidz outage [5120292,5124552] length 4261 outage [5124554,5124554] length 1 outage [5124556,5124558] length 3 outage [5124560,5124562] length 3 outage [5124564,5124564] length 1 outage [5124566,5124568] length 3 outage [5124570,5124572] length 3 outage [5124574,5124574] length 1 outage [5124576,5124582] length 7 outage [5124584,5124584] length 1 outage [5124586,5124588] length 3 outage [5124590,5124592] length 3 outage [5124594,5124594] length 1 outage [5124596,5124598] length 3 outage [5124600,5124602] length 3 outage [5124604,5124604] length 1 outage [5124606,5124608] length 3 outage [5124610,5124612] length 3 outage [5124614,5124614] length 1 outage [5124616,5124618] length 3 outage [5124620,5124624] length 5 outage [5124626,5124628] length 3 outage [5124630,5124632] length 3 outage [5124634,5124634] length 1 outage [5124636,5124638] length 3 outage [5124640,5124642] length 3 outage [5124644,5124648] length 5 outage [5124650,5124652] length 3 outage [5124654,5124654] length 1 outage [5124656,5124658] length 3 outage [5124660,5124662] length 3 outage [5124664,5124664] length 1 outage [5124666,5124668] length 3 outage [5124670,5124672] length 3 outage [5124674,5124674] length 1 outage [5124676,5124678] length 3 outage [5124680,5124682] length 3 outage [5124684,5124684] length 1 outage [5124686,5124688] length 3 outage [5124690,5124692] length 3 outage [5124694,5124694] length 1 outage [5124696,5124710] length 15 outage [5124712,5124712] length 1 outage [5124714,5124721] length 8 outage [5124723,5124723] length 1 outage [5124725,5124727] length 3 outage [5124729,5124729] length 1 outage [5124731,5124731] length 1 outage [5124733,5124737] length 5 outage [5124739,5124739] length 1 outage [5124741,5124747] length 7 outage [5124749,5124749] length 1 outage [5124751,5124751] length 1 outage [5124753,5124753] length 1 outage [5124755,5124757] length 3 outage [5124759,5124768] length 10 outage [5124770,5124770] length 1 outage [5124772,5124772] length 1 outage [5124774,5124774] length 1 outage [5124776,5124784] length 9 outage [5124786,5124791] length 6 outage [5124793,5124793] length 1 outage [5124795,5124795] length 1 outage [5124797,5124797] length 1 outage [5124799,5124799] length 1 outage [5124801,5124801] length 1 outage [5124803,5124803] length 1 outage [5124805,5124805] length 1 outage [5124807,5124807] length 1 outage [5124809,5124809] length 1 outage [5124811,5124811] length 1 outage [5124813,5124813] length 1 outage [5124815,5124815] length 1 outage [5124817,5124817] length 1 outage [5124819,5124819] length 1 outage [5124821,5124821] length 1 outage [5124823,5124823] length 1 outage [5124825,5124825] length 1 outage [5124827,5124827] length 1 outage [5124829,5124829] length 1 outage [5124831,5124831] length 1 outage [5124833,5124833] length 1 outage [5124835,5124835] length 1 outage [5124837,5124837] length 1 outage [5124839,5124839] length 1 outage [5124841,5124841] length 1 outage [5124843,5124845] length 3 outage [5124847,5124847] length 1 outage [5124849,5124849] length 1 outage [5124851,5124851] length 1 outage [5124853,5124855] length 3 outage [5124857,5124857] length 1 outage [5124859,5124859] length 1 outage [5124861,5124861] length 1 outage [5124863,5124865] length 3 outage [5124867,5124867] length 1 outage [5124869,5124869] length 1 outage [5124871,5124871] length 1 outage [5124873,5124875] length 3 outage [5124877,5124877] length 1 outage [5124879,5124879] length 1 outage [5124881,5124881] length 1 outage [5124883,5124885] length 3 outage [5124887,5124887] length 1 outage [5124889,5124889] length 1 outage [5124891,5124891] length 1 outage [5124893,5124895] length 3 outage [5124897,5124897] length 1 outage [5124899,5124899] length 1 outage [5124901,5124901] length 1 outage [5124903,5124905] length 3 outage [5124907,5124907] length 1 outage [5124909,5124911] length 3 outage [5124913,5124915] length 3 outage [5124917,5124917] length 1 outage [5124919,5124919] length 1 outage [5124921,5124921] length 1 outage [5124923,5124925] length 3 outage [5124927,5124927] length 1 outage [5124929,5124929] length 1 outage [5124931,5124931] length 1 outage [5124933,5124935] length 3 outage [5124937,5124937] length 1 outage [5124939,5124939] length 1 outage [5124941,5124941] length 1 outage [5124943,5124945] length 3 outage [5124947,5124947] length 1 outage [5124949,5124949] length 1 outage [5124951,5124951] length 1 outage [5124953,5124955] length 3 outage [5124957,5124957] length 1 outage [5124959,5124959] length 1 outage [5124961,5124961] length 1 outage [5124963,5124965] length 3 outage [5124967,5124967] length 1 outage [5124969,5124969] length 1 outage [5124971,5124971] length 1 outage [5124973,5124975] length 3 outage [5124977,5124977] length 1 outage [5124979,5124979] length 1 outage [5124981,5124981] length 1 outage [5124983,5124985] length 3 outage [5124987,5124987] length 1 outage [5124989,5124989] length 1 outage [5124991,5124991] length 1 outage [5124993,5124995] length 3 outage [5124997,5124997] length 1 outage [5124999,5124999] length 1 outage [5125001,5125001] length 1 outage [5125003,5125005] length 3 outage [5125007,5125007] length 1 outage [5125009,5125009] length 1 outage [5125011,5125011] length 1 outage [5125013,5125015] length 3 outage [5125017,5125017] length 1 outage [5125019,5125019] length 1 outage [5125021,5125021] length 1 outage [5125023,5125025] length 3 outage [5125027,5125027] length 1 outage [5125029,5125029] length 1 outage [5125031,5125031] length 1 outage [5125033,5125035] length 3 outage [5125037,5125037] length 1 outage [5125039,5125039] length 1 outage [5125041,5125041] length 1 outage [5125043,5125045] length 3 outage [5125047,5125047] length 1 outage [5125049,5125049] length 1 outage [5125051,5125051] length 1 outage [5125053,5125055] length 3 outage [5125057,5125057] length 1 outage [5125059,5125059] length 1 outage [5125061,5125061] length 1 outage [5125063,5125065] length 3 outage [5125067,5125067] length 1 outage [5125069,5125069] length 1 outage [5125071,5125071] length 1 outage [5125073,5125075] length 3 outage [5125077,5125077] length 1 outage [5125079,5125079] length 1 outage [5125081,5125081] length 1 outage [5125083,5125085] length 3 outage [5125087,5125087] length 1 outage [5125089,5125089] length 1 outage [5125091,5125091] length 1 outage [5125093,5125095] length 3 outage [5125097,5125097] length 1 outage [5125099,5125099] length 1 outage [5125101,5125101] length 1 outage [5125103,5125105] length 3 outage [5125107,5125107] length 1 outage [5125109,5125109] length 1 outage [5125111,5125111] length 1 outage [5125113,5125115] length 3 outage [5125117,5125117] length 1 outage [5125119,5125119] length 1 outage [5125121,5125121] length 1 outage [5125123,5125125] length 3 outage [5125127,5125127] length 1 outage [5125129,5125129] length 1 outage [5125131,5125131] length 1 outage [5125133,5125135] length 3 outage [5125137,5125137] length 1 outage [5125139,5125139] length 1 outage [5125141,5125141] length 1 outage [5125143,5125145] length 3 outage [5125147,5125147] length 1 outage [5125149,5125149] length 1 outage [5125151,5125151] length 1 outage [5125153,5125154] length 2 outage [5125156,5125156] length 1 outage [5125158,5125160] length 3 outage [5125162,5125162] length 1 outage [5125164,5125164] length 1 outage [5125166,5125166] length 1 outage [5125168,5125170] length 3 outage [5125172,5125172] length 1 outage [5125174,5125175] length 2 outage [5125177,5125177] length 1 outage [5125179,5125179] length 1 outage [5125181,5125181] length 1 outage [5125183,5125185] length 3 outage [5125187,5125187] length 1 outage [5125189,5125189] length 1 outage [5125191,5125191] length 1 outage [5125193,5125195] length 3 outage [5125197,5125197] length 1 outage [5125199,5125199] length 1 outage [5125201,5125201] length 1 outage [5125203,5125205] length 3 outage [5125207,5125207] length 1 outage [5125209,5125209] length 1 outage [5125211,5125211] length 1 outage [5125213,5125215] length 3 outage [5125217,5125217] length 1 outage [5125219,5125219] length 1 outage [5125221,5125221] length 1 outage [5125223,5125225] length 3 outage [5125227,5125227] length 1 outage [5125229,5125229] length 1 outage [5125231,5125231] length 1 outage [5125233,5125235] length 3 outage [5125237,5125237] length 1 outage [5125239,5125239] length 1 outage [5125241,5125241] length 1 outage [5125243,5125245] length 3 outage [5125247,5125249] length 3 outage [5125251,5125251] length 1 outage [5125253,5125255] length 3 outage [5125257,5125257] length 1 outage [5125259,5125259] length 1 outage [5125261,5125261] length 1 outage [5125263,5125265] length 3 outage [5125267,5125267] length 1 outage [5125269,5125269] length 1 outage [5125271,5125271] length 1 outage [5125273,5125275] length 3 outage [5125277,5125277] length 1 outage [5125279,5125279] length 1 outage [5125281,5125281] length 1 outage [5125283,5125285] length 3 outage [5125287,5125287] length 1 outage [5125289,5125289] length 1 outage [5125291,5125291] length 1 outage [5125293,5125295] length 3 outage [5125297,5125297] length 1 outage [5125299,5125299] length 1 outage [5125301,5125301] length 1 outage [5125303,5125305] length 3 outage [5125307,5125307] length 1 outage [5125309,5125309] length 1 outage [5125311,5125311] length 1 outage [5125313,5125315] length 3 outage [5125317,5125317] length 1 outage [5125319,5125319] length 1 outage [5125321,5125321] length 1 outage [5125323,5125325] length 3 outage [5125327,5125327] length 1 outage [5125329,5125329] length 1 outage [5125331,5125331] length 1 outage [5125333,5125335] length 3 outage [5125337,5125337] length 1 outage [5125339,5125339] length 1 outage [5125341,5125341] length 1 outage [5125343,5125345] length 3 outage [5125347,5125347] length 1 outage [5125349,5125349] length 1 outage [5125351,5125351] length 1 outage [5125353,5125355] length 3 outage [5125357,5125357] length 1 outage [5125359,5125359] length 1 outage [5125361,5125361] length 1 outage [5125363,5125365] length 3 outage [5125367,5125367] length 1 outage [5125369,5125369] length 1 outage [5125371,5125371] length 1 outage [5125373,5125375] length 3 outage [5125377,5125377] length 1 outage [5125379,5125379] length 1 outage [5125381,5125381] length 1 outage [5125383,5125385] length 3 outage [5125387,5125387] length 1 outage [5125389,5125389] length 1 outage [5125391,5125391] length 1 outage [5125393,5125395] length 3 outage [5125397,5125397] length 1 outage [5125399,5125399] length 1 outage [5125401,5125401] length 1 outage [5125403,5125405] length 3 outage [5125407,5125407] length 1 outage [5125409,5125409] length 1 outage [5125411,5125411] length 1 outage [5125413,5125415] length 3 outage [5125417,5125417] length 1 outage [5125419,5125419] length 1 outage [5125421,5125421] length 1 outage [5125423,5125440] length 18 outage [5125442,5125444] length 3 outage [5125446,5125446] length 1 outage [5125448,5125450] length 3 outage [5125452,5125452] length 1 outage [5125455,5125456] length 2 outage [5125466,5125468] length 3 outage [5125470,5125472] length 3 outage [5125474,5125474] length 1 outage [5125476,5125479] length 4 outage [5125481,5125483] length 3 outage [5125485,5125485] length 1 outage [5125487,5125496] length 10 outage [5125498,5125498] length 1 outage [5125500,5125500] length 1 outage [5125502,5125502] length 1 outage [5125504,5125504] length 1 outage [5125506,5125506] length 1 outage [5125508,5125508] length 1 outage [5125510,5125510] length 1 outage [5125512,5125512] length 1 outage [5125514,5125514] length 1 outage [5125516,5125516] length 1 outage [5125518,5125518] length 1 outage [5125520,5125520] length 1 outage [5125522,5125522] length 1 outage [5125524,5125524] length 1 outage [5125526,5125526] length 1 outage [5125528,5125528] length 1 outage [5125530,5125530] length 1 outage [5125532,5125532] length 1 outage [5125534,5125537] length 4 /dev/label/disk01 /dev/label/disk02 /dev/label/disk03 replacing outage [5120292,5124552] length 4261 outage [5124554,5124554] length 1 outage [5124556,5124558] length 3 outage [5124560,5124562] length 3 outage [5124564,5124564] length 1 outage [5124566,5124568] length 3 outage [5124570,5124572] length 3 outage [5124574,5124574] length 1 outage [5124576,5124582] length 7 outage [5124584,5124584] length 1 outage [5124586,5124588] length 3 outage [5124590,5124592] length 3 outage [5124594,5124594] length 1 outage [5124596,5124598] length 3 outage [5124600,5124602] length 3 outage [5124604,5124604] length 1 outage [5124606,5124608] length 3 outage [5124610,5124612] length 3 outage [5124614,5124614] length 1 outage [5124616,5124618] length 3 outage [5124620,5124624] length 5 outage [5124626,5124628] length 3 outage [5124630,5124632] length 3 outage [5124634,5124634] length 1 outage [5124636,5124638] length 3 outage [5124640,5124642] length 3 outage [5124644,5124648] length 5 outage [5124650,5124652] length 3 outage [5124654,5124654] length 1 outage [5124656,5124658] length 3 outage [5124660,5124662] length 3 outage [5124664,5124664] length 1 outage [5124666,5124668] length 3 outage [5124670,5124672] length 3 outage [5124674,5124674] length 1 outage [5124676,5124678] length 3 outage [5124680,5124682] length 3 outage [5124684,5124684] length 1 outage [5124686,5124688] length 3 outage [5124690,5124692] length 3 outage [5124694,5124694] length 1 outage [5124696,5124710] length 15 outage [5124712,5124712] length 1 outage [5124714,5124721] length 8 outage [5124723,5124723] length 1 outage [5124725,5124727] length 3 outage [5124729,5124729] length 1 outage [5124731,5124731] length 1 outage [5124733,5124737] length 5 outage [5124739,5124739] length 1 outage [5124741,5124747] length 7 outage [5124749,5124749] length 1 outage [5124751,5124751] length 1 outage [5124753,5124753] length 1 outage [5124755,5124757] length 3 outage [5124759,5124768] length 10 outage [5124770,5124770] length 1 outage [5124772,5124772] length 1 outage [5124774,5124774] length 1 outage [5124776,5124784] length 9 outage [5124786,5124791] length 6 outage [5124793,5124793] length 1 outage [5124795,5124795] length 1 outage [5124797,5124797] length 1 outage [5124799,5124799] length 1 outage [5124801,5124801] length 1 outage [5124803,5124803] length 1 outage [5124805,5124805] length 1 outage [5124807,5124807] length 1 outage [5124809,5124809] length 1 outage [5124811,5124811] length 1 outage [5124813,5124813] length 1 outage [5124815,5124815] length 1 outage [5124817,5124817] length 1 outage [5124819,5124819] length 1 outage [5124821,5124821] length 1 outage [5124823,5124823] length 1 outage [5124825,5124825] length 1 outage [5124827,5124827] length 1 outage [5124829,5124829] length 1 outage [5124831,5124831] length 1 outage [5124833,5124833] length 1 outage [5124835,5124835] length 1 outage [5124837,5124837] length 1 outage [5124839,5124839] length 1 outage [5124841,5124841] length 1 outage [5124843,5124845] length 3 outage [5124847,5124847] length 1 outage [5124849,5124849] length 1 outage [5124851,5124851] length 1 outage [5124853,5124855] length 3 outage [5124857,5124857] length 1 outage [5124859,5124859] length 1 outage [5124861,5124861] length 1 outage [5124863,5124865] length 3 outage [5124867,5124867] length 1 outage [5124869,5124869] length 1 outage [5124871,5124871] length 1 outage [5124873,5124875] length 3 outage [5124877,5124877] length 1 outage [5124879,5124879] length 1 outage [5124881,5124881] length 1 outage [5124883,5124885] length 3 outage [5124887,5124887] length 1 outage [5124889,5124889] length 1 outage [5124891,5124891] length 1 outage [5124893,5124895] length 3 outage [5124897,5124897] length 1 outage [5124899,5124899] length 1 outage [5124901,5124901] length 1 outage [5124903,5124905] length 3 outage [5124907,5124907] length 1 outage [5124909,5124911] length 3 outage [5124913,5124915] length 3 outage [5124917,5124917] length 1 outage [5124919,5124919] length 1 outage [5124921,5124921] length 1 outage [5124923,5124925] length 3 outage [5124927,5124927] length 1 outage [5124929,5124929] length 1 outage [5124931,5124931] length 1 outage [5124933,5124935] length 3 outage [5124937,5124937] length 1 outage [5124939,5124939] length 1 outage [5124941,5124941] length 1 outage [5124943,5124945] length 3 outage [5124947,5124947] length 1 outage [5124949,5124949] length 1 outage [5124951,5124951] length 1 outage [5124953,5124955] length 3 outage [5124957,5124957] length 1 outage [5124959,5124959] length 1 outage [5124961,5124961] length 1 outage [5124963,5124965] length 3 outage [5124967,5124967] length 1 outage [5124969,5124969] length 1 outage [5124971,5124971] length 1 outage [5124973,5124975] length 3 outage [5124977,5124977] length 1 outage [5124979,5124979] length 1 outage [5124981,5124981] length 1 outage [5124983,5124985] length 3 outage [5124987,5124987] length 1 outage [5124989,5124989] length 1 outage [5124991,5124991] length 1 outage [5124993,5124995] length 3 outage [5124997,5124997] length 1 outage [5124999,5124999] length 1 outage [5125001,5125001] length 1 outage [5125003,5125005] length 3 outage [5125007,5125007] length 1 outage [5125009,5125009] length 1 outage [5125011,5125011] length 1 outage [5125013,5125015] length 3 outage [5125017,5125017] length 1 outage [5125019,5125019] length 1 outage [5125021,5125021] length 1 outage [5125023,5125025] length 3 outage [5125027,5125027] length 1 outage [5125029,5125029] length 1 outage [5125031,5125031] length 1 outage [5125033,5125035] length 3 outage [5125037,5125037] length 1 outage [5125039,5125039] length 1 outage [5125041,5125041] length 1 outage [5125043,5125045] length 3 outage [5125047,5125047] length 1 outage [5125049,5125049] length 1 outage [5125051,5125051] length 1 outage [5125053,5125055] length 3 outage [5125057,5125057] length 1 outage [5125059,5125059] length 1 outage [5125061,5125061] length 1 outage [5125063,5125065] length 3 outage [5125067,5125067] length 1 outage [5125069,5125069] length 1 outage [5125071,5125071] length 1 outage [5125073,5125075] length 3 outage [5125077,5125077] length 1 outage [5125079,5125079] length 1 outage [5125081,5125081] length 1 outage [5125083,5125085] length 3 outage [5125087,5125087] length 1 outage [5125089,5125089] length 1 outage [5125091,5125091] length 1 outage [5125093,5125095] length 3 outage [5125097,5125097] length 1 outage [5125099,5125099] length 1 outage [5125101,5125101] length 1 outage [5125103,5125105] length 3 outage [5125107,5125107] length 1 outage [5125109,5125109] length 1 outage [5125111,5125111] length 1 outage [5125113,5125115] length 3 outage [5125117,5125117] length 1 outage [5125119,5125119] length 1 outage [5125121,5125121] length 1 outage [5125123,5125125] length 3 outage [5125127,5125127] length 1 outage [5125129,5125129] length 1 outage [5125131,5125131] length 1 outage [5125133,5125135] length 3 outage [5125137,5125137] length 1 outage [5125139,5125139] length 1 outage [5125141,5125141] length 1 outage [5125143,5125145] length 3 outage [5125147,5125147] length 1 outage [5125149,5125149] length 1 outage [5125151,5125151] length 1 outage [5125153,5125154] length 2 outage [5125156,5125156] length 1 outage [5125158,5125160] length 3 outage [5125162,5125162] length 1 outage [5125164,5125164] length 1 outage [5125166,5125166] length 1 outage [5125168,5125170] length 3 outage [5125172,5125172] length 1 outage [5125174,5125175] length 2 outage [5125177,5125177] length 1 outage [5125179,5125179] length 1 outage [5125181,5125181] length 1 outage [5125183,5125185] length 3 outage [5125187,5125187] length 1 outage [5125189,5125189] length 1 outage [5125191,5125191] length 1 outage [5125193,5125195] length 3 outage [5125197,5125197] length 1 outage [5125199,5125199] length 1 outage [5125201,5125201] length 1 outage [5125203,5125205] length 3 outage [5125207,5125207] length 1 outage [5125209,5125209] length 1 outage [5125211,5125211] length 1 outage [5125213,5125215] length 3 outage [5125217,5125217] length 1 outage [5125219,5125219] length 1 outage [5125221,5125221] length 1 outage [5125223,5125225] length 3 outage [5125227,5125227] length 1 outage [5125229,5125229] length 1 outage [5125231,5125231] length 1 outage [5125233,5125235] length 3 outage [5125237,5125237] length 1 outage [5125239,5125239] length 1 outage [5125241,5125241] length 1 outage [5125243,5125245] length 3 outage [5125247,5125249] length 3 outage [5125251,5125251] length 1 outage [5125253,5125255] length 3 outage [5125257,5125257] length 1 outage [5125259,5125259] length 1 outage [5125261,5125261] length 1 outage [5125263,5125265] length 3 outage [5125267,5125267] length 1 outage [5125269,5125269] length 1 outage [5125271,5125271] length 1 outage [5125273,5125275] length 3 outage [5125277,5125277] length 1 outage [5125279,5125279] length 1 outage [5125281,5125281] length 1 outage [5125283,5125285] length 3 outage [5125287,5125287] length 1 outage [5125289,5125289] length 1 outage [5125291,5125291] length 1 outage [5125293,5125295] length 3 outage [5125297,5125297] length 1 outage [5125299,5125299] length 1 outage [5125301,5125301] length 1 outage [5125303,5125305] length 3 outage [5125307,5125307] length 1 outage [5125309,5125309] length 1 outage [5125311,5125311] length 1 outage [5125313,5125315] length 3 outage [5125317,5125317] length 1 outage [5125319,5125319] length 1 outage [5125321,5125321] length 1 outage [5125323,5125325] length 3 outage [5125327,5125327] length 1 outage [5125329,5125329] length 1 outage [5125331,5125331] length 1 outage [5125333,5125335] length 3 outage [5125337,5125337] length 1 outage [5125339,5125339] length 1 outage [5125341,5125341] length 1 outage [5125343,5125345] length 3 outage [5125347,5125347] length 1 outage [5125349,5125349] length 1 outage [5125351,5125351] length 1 outage [5125353,5125355] length 3 outage [5125357,5125357] length 1 outage [5125359,5125359] length 1 outage [5125361,5125361] length 1 outage [5125363,5125365] length 3 outage [5125367,5125367] length 1 outage [5125369,5125369] length 1 outage [5125371,5125371] length 1 outage [5125373,5125375] length 3 outage [5125377,5125377] length 1 outage [5125379,5125379] length 1 outage [5125381,5125381] length 1 outage [5125383,5125385] length 3 outage [5125387,5125387] length 1 outage [5125389,5125389] length 1 outage [5125391,5125391] length 1 outage [5125393,5125395] length 3 outage [5125397,5125397] length 1 outage [5125399,5125399] length 1 outage [5125401,5125401] length 1 outage [5125403,5125405] length 3 outage [5125407,5125407] length 1 outage [5125409,5125409] length 1 outage [5125411,5125411] length 1 outage [5125413,5125415] length 3 outage [5125417,5125417] length 1 outage [5125419,5125419] length 1 outage [5125421,5125421] length 1 outage [5125423,5125440] length 18 outage [5125442,5125444] length 3 outage [5125446,5125446] length 1 outage [5125448,5125450] length 3 outage [5125452,5125452] length 1 outage [5125455,5125456] length 2 outage [5125466,5125468] length 3 outage [5125470,5125472] length 3 outage [5125474,5125474] length 1 outage [5125476,5125479] length 4 outage [5125481,5125483] length 3 outage [5125485,5125485] length 1 outage [5125487,5125496] length 10 outage [5125498,5125498] length 1 outage [5125500,5125500] length 1 outage [5125502,5125502] length 1 outage [5125504,5125504] length 1 outage [5125506,5125506] length 1 outage [5125508,5125508] length 1 outage [5125510,5125510] length 1 outage [5125512,5125512] length 1 outage [5125514,5125514] length 1 outage [5125516,5125516] length 1 outage [5125518,5125518] length 1 outage [5125520,5125520] length 1 outage [5125522,5125522] length 1 outage [5125524,5125524] length 1 outage [5125526,5125526] length 1 outage [5125528,5125528] length 1 outage [5125530,5125530] length 1 outage [5125532,5125532] length 1 outage [5125534,5125537] length 4 /dev/label/disk04/old outage [5120292,5124552] length 4261 outage [5124554,5124554] length 1 outage [5124556,5124558] length 3 outage [5124560,5124562] length 3 outage [5124564,5124564] length 1 outage [5124566,5124568] length 3 outage [5124570,5124572] length 3 outage [5124574,5124574] length 1 outage [5124576,5124582] length 7 outage [5124584,5124584] length 1 outage [5124586,5124588] length 3 outage [5124590,5124592] length 3 outage [5124594,5124594] length 1 outage [5124596,5124598] length 3 outage [5124600,5124602] length 3 outage [5124604,5124604] length 1 outage [5124606,5124608] length 3 outage [5124610,5124612] length 3 outage [5124614,5124614] length 1 outage [5124616,5124618] length 3 outage [5124620,5124624] length 5 outage [5124626,5124628] length 3 outage [5124630,5124632] length 3 outage [5124634,5124634] length 1 outage [5124636,5124638] length 3 outage [5124640,5124642] length 3 outage [5124644,5124648] length 5 outage [5124650,5124652] length 3 outage [5124654,5124654] length 1 outage [5124656,5124658] length 3 outage [5124660,5124662] length 3 outage [5124664,5124664] length 1 outage [5124666,5124668] length 3 outage [5124670,5124672] length 3 outage [5124674,5124674] length 1 outage [5124676,5124678] length 3 outage [5124680,5124682] length 3 outage [5124684,5124684] length 1 outage [5124686,5124688] length 3 outage [5124690,5124692] length 3 outage [5124694,5124694] length 1 outage [5124696,5124710] length 15 outage [5124712,5124712] length 1 outage [5124714,5124721] length 8 outage [5124723,5124723] length 1 outage [5124725,5124727] length 3 outage [5124729,5124729] length 1 outage [5124731,5124731] length 1 outage [5124733,5124737] length 5 outage [5124739,5124739] length 1 outage [5124741,5124747] length 7 outage [5124749,5124749] length 1 outage [5124751,5124751] length 1 outage [5124753,5124753] length 1 outage [5124755,5124757] length 3 outage [5124759,5124768] length 10 outage [5124770,5124770] length 1 outage [5124772,5124772] length 1 outage [5124774,5124774] length 1 outage [5124776,5124784] length 9 outage [5124786,5124791] length 6 outage [5124793,5124793] length 1 outage [5124795,5124795] length 1 outage [5124797,5124797] length 1 outage [5124799,5124799] length 1 outage [5124801,5124801] length 1 outage [5124803,5124803] length 1 outage [5124805,5124805] length 1 outage [5124807,5124807] length 1 outage [5124809,5124809] length 1 outage [5124811,5124811] length 1 outage [5124813,5124813] length 1 outage [5124815,5124815] length 1 outage [5124817,5124817] length 1 outage [5124819,5124819] length 1 outage [5124821,5124821] length 1 outage [5124823,5124823] length 1 outage [5124825,5124825] length 1 outage [5124827,5124827] length 1 outage [5124829,5124829] length 1 outage [5124831,5124831] length 1 outage [5124833,5124833] length 1 outage [5124835,5124835] length 1 outage [5124837,5124837] length 1 outage [5124839,5124839] length 1 outage [5124841,5124841] length 1 outage [5124843,5124845] length 3 outage [5124847,5124847] length 1 outage [5124849,5124849] length 1 outage [5124851,5124851] length 1 outage [5124853,5124855] length 3 outage [5124857,5124857] length 1 outage [5124859,5124859] length 1 outage [5124861,5124861] length 1 outage [5124863,5124865] length 3 outage [5124867,5124867] length 1 outage [5124869,5124869] length 1 outage [5124871,5124871] length 1 outage [5124873,5124875] length 3 outage [5124877,5124877] length 1 outage [5124879,5124879] length 1 outage [5124881,5124881] length 1 outage [5124883,5124885] length 3 outage [5124887,5124887] length 1 outage [5124889,5124889] length 1 outage [5124891,5124891] length 1 outage [5124893,5124895] length 3 outage [5124897,5124897] length 1 outage [5124899,5124899] length 1 outage [5124901,5124901] length 1 outage [5124903,5124905] length 3 outage [5124907,5124907] length 1 outage [5124909,5124911] length 3 outage [5124913,5124915] length 3 outage [5124917,5124917] length 1 outage [5124919,5124919] length 1 outage [5124921,5124921] length 1 outage [5124923,5124925] length 3 outage [5124927,5124927] length 1 outage [5124929,5124929] length 1 outage [5124931,5124931] length 1 outage [5124933,5124935] length 3 outage [5124937,5124937] length 1 outage [5124939,5124939] length 1 outage [5124941,5124941] length 1 outage [5124943,5124945] length 3 outage [5124947,5124947] length 1 outage [5124949,5124949] length 1 outage [5124951,5124951] length 1 outage [5124953,5124955] length 3 outage [5124957,5124957] length 1 outage [5124959,5124959] length 1 outage [5124961,5124961] length 1 outage [5124963,5124965] length 3 outage [5124967,5124967] length 1 outage [5124969,5124969] length 1 outage [5124971,5124971] length 1 outage [5124973,5124975] length 3 outage [5124977,5124977] length 1 outage [5124979,5124979] length 1 outage [5124981,5124981] length 1 outage [5124983,5124985] length 3 outage [5124987,5124987] length 1 outage [5124989,5124989] length 1 outage [5124991,5124991] length 1 outage [5124993,5124995] length 3 outage [5124997,5124997] length 1 outage [5124999,5124999] length 1 outage [5125001,5125001] length 1 outage [5125003,5125005] length 3 outage [5125007,5125007] length 1 outage [5125009,5125009] length 1 outage [5125011,5125011] length 1 outage [5125013,5125015] length 3 outage [5125017,5125017] length 1 outage [5125019,5125019] length 1 outage [5125021,5125021] length 1 outage [5125023,5125025] length 3 outage [5125027,5125027] length 1 outage [5125029,5125029] length 1 outage [5125031,5125031] length 1 outage [5125033,5125035] length 3 outage [5125037,5125037] length 1 outage [5125039,5125039] length 1 outage [5125041,5125041] length 1 outage [5125043,5125045] length 3 outage [5125047,5125047] length 1 outage [5125049,5125049] length 1 outage [5125051,5125051] length 1 outage [5125053,5125055] length 3 outage [5125057,5125057] length 1 outage [5125059,5125059] length 1 outage [5125061,5125061] length 1 outage [5125063,5125065] length 3 outage [5125067,5125067] length 1 outage [5125069,5125069] length 1 outage [5125071,5125071] length 1 outage [5125073,5125075] length 3 outage [5125077,5125077] length 1 outage [5125079,5125079] length 1 outage [5125081,5125081] length 1 outage [5125083,5125085] length 3 outage [5125087,5125087] length 1 outage [5125089,5125089] length 1 outage [5125091,5125091] length 1 outage [5125093,5125095] length 3 outage [5125097,5125097] length 1 outage [5125099,5125099] length 1 outage [5125101,5125101] length 1 outage [5125103,5125105] length 3 outage [5125107,5125107] length 1 outage [5125109,5125109] length 1 outage [5125111,5125111] length 1 outage [5125113,5125115] length 3 outage [5125117,5125117] length 1 outage [5125119,5125119] length 1 outage [5125121,5125121] length 1 outage [5125123,5125125] length 3 outage [5125127,5125127] length 1 outage [5125129,5125129] length 1 outage [5125131,5125131] length 1 outage [5125133,5125135] length 3 outage [5125137,5125137] length 1 outage [5125139,5125139] length 1 outage [5125141,5125141] length 1 outage [5125143,5125145] length 3 outage [5125147,5125147] length 1 outage [5125149,5125149] length 1 outage [5125151,5125151] length 1 outage [5125153,5125154] length 2 outage [5125156,5125156] length 1 outage [5125158,5125160] length 3 outage [5125162,5125162] length 1 outage [5125164,5125164] length 1 outage [5125166,5125166] length 1 outage [5125168,5125170] length 3 outage [5125172,5125172] length 1 outage [5125174,5125175] length 2 outage [5125177,5125177] length 1 outage [5125179,5125179] length 1 outage [5125181,5125181] length 1 outage [5125183,5125185] length 3 outage [5125187,5125187] length 1 outage [5125189,5125189] length 1 outage [5125191,5125191] length 1 outage [5125193,5125195] length 3 outage [5125197,5125197] length 1 outage [5125199,5125199] length 1 outage [5125201,5125201] length 1 outage [5125203,5125205] length 3 outage [5125207,5125207] length 1 outage [5125209,5125209] length 1 outage [5125211,5125211] length 1 outage [5125213,5125215] length 3 outage [5125217,5125217] length 1 outage [5125219,5125219] length 1 outage [5125221,5125221] length 1 outage [5125223,5125225] length 3 outage [5125227,5125227] length 1 outage [5125229,5125229] length 1 outage [5125231,5125231] length 1 outage [5125233,5125235] length 3 outage [5125237,5125237] length 1 outage [5125239,5125239] length 1 outage [5125241,5125241] length 1 outage [5125243,5125245] length 3 outage [5125247,5125249] length 3 outage [5125251,5125251] length 1 outage [5125253,5125255] length 3 outage [5125257,5125257] length 1 outage [5125259,5125259] length 1 outage [5125261,5125261] length 1 outage [5125263,5125265] length 3 outage [5125267,5125267] length 1 outage [5125269,5125269] length 1 outage [5125271,5125271] length 1 outage [5125273,5125275] length 3 outage [5125277,5125277] length 1 outage [5125279,5125279] length 1 outage [5125281,5125281] length 1 outage [5125283,5125285] length 3 outage [5125287,5125287] length 1 outage [5125289,5125289] length 1 outage [5125291,5125291] length 1 outage [5125293,5125295] length 3 outage [5125297,5125297] length 1 outage [5125299,5125299] length 1 outage [5125301,5125301] length 1 outage [5125303,5125305] length 3 outage [5125307,5125307] length 1 outage [5125309,5125309] length 1 outage [5125311,5125311] length 1 outage [5125313,5125315] length 3 outage [5125317,5125317] length 1 outage [5125319,5125319] length 1 outage [5125321,5125321] length 1 outage [5125323,5125325] length 3 outage [5125327,5125327] length 1 outage [5125329,5125329] length 1 outage [5125331,5125331] length 1 outage [5125333,5125335] length 3 outage [5125337,5125337] length 1 outage [5125339,5125339] length 1 outage [5125341,5125341] length 1 outage [5125343,5125345] length 3 outage [5125347,5125347] length 1 outage [5125349,5125349] length 1 outage [5125351,5125351] length 1 outage [5125353,5125355] length 3 outage [5125357,5125357] length 1 outage [5125359,5125359] length 1 outage [5125361,5125361] length 1 outage [5125363,5125365] length 3 outage [5125367,5125367] length 1 outage [5125369,5125369] length 1 outage [5125371,5125371] length 1 outage [5125373,5125375] length 3 outage [5125377,5125377] length 1 outage [5125379,5125379] length 1 outage [5125381,5125381] length 1 outage [5125383,5125385] length 3 outage [5125387,5125387] length 1 outage [5125389,5125389] length 1 outage [5125391,5125391] length 1 outage [5125393,5125395] length 3 outage [5125397,5125397] length 1 outage [5125399,5125399] length 1 outage [5125401,5125401] length 1 outage [5125403,5125405] length 3 outage [5125407,5125407] length 1 outage [5125409,5125409] length 1 outage [5125411,5125411] length 1 outage [5125413,5125415] length 3 outage [5125417,5125417] length 1 outage [5125419,5125419] length 1 outage [5125421,5125421] length 1 outage [5125423,5125440] length 18 outage [5125442,5125444] length 3 outage [5125446,5125446] length 1 outage [5125448,5125450] length 3 outage [5125452,5125452] length 1 outage [5125455,5125456] length 2 outage [5125466,5125468] length 3 outage [5125470,5125472] length 3 outage [5125474,5125474] length 1 outage [5125476,5125479] length 4 outage [5125481,5125483] length 3 outage [5125485,5125485] length 1 outage [5125487,5125496] length 10 outage [5125498,5125498] length 1 outage [5125500,5125500] length 1 outage [5125502,5125502] length 1 outage [5125504,5125504] length 1 outage [5125506,5125506] length 1 outage [5125508,5125508] length 1 outage [5125510,5125510] length 1 outage [5125512,5125512] length 1 outage [5125514,5125514] length 1 outage [5125516,5125516] length 1 outage [5125518,5125518] length 1 outage [5125520,5125520] length 1 outage [5125522,5125522] length 1 outage [5125524,5125524] length 1 outage [5125526,5125526] length 1 outage [5125528,5125528] length 1 outage [5125530,5125530] length 1 outage [5125532,5125532] length 1 outage [5125534,5125537] length 4 /dev/label/disk04 outage [5120292,5124552] length 4261 outage [5124554,5124554] length 1 outage [5124556,5124558] length 3 outage [5124560,5124562] length 3 outage [5124564,5124564] length 1 outage [5124566,5124568] length 3 outage [5124570,5124572] length 3 outage [5124574,5124574] length 1 outage [5124576,5124582] length 7 outage [5124584,5124584] length 1 outage [5124586,5124588] length 3 outage [5124590,5124592] length 3 outage [5124594,5124594] length 1 outage [5124596,5124598] length 3 outage [5124600,5124602] length 3 outage [5124604,5124604] length 1 outage [5124606,5124608] length 3 outage [5124610,5124612] length 3 outage [5124614,5124614] length 1 outage [5124616,5124618] length 3 outage [5124620,5124624] length 5 outage [5124626,5124628] length 3 outage [5124630,5124632] length 3 outage [5124634,5124634] length 1 outage [5124636,5124638] length 3 outage [5124640,5124642] length 3 outage [5124644,5124648] length 5 outage [5124650,5124652] length 3 outage [5124654,5124654] length 1 outage [5124656,5124658] length 3 outage [5124660,5124662] length 3 outage [5124664,5124664] length 1 outage [5124666,5124668] length 3 outage [5124670,5124672] length 3 outage [5124674,5124674] length 1 outage [5124676,5124678] length 3 outage [5124680,5124682] length 3 outage [5124684,5124684] length 1 outage [5124686,5124688] length 3 outage [5124690,5124692] length 3 outage [5124694,5124694] length 1 outage [5124696,5124710] length 15 outage [5124712,5124712] length 1 outage [5124714,5124721] length 8 outage [5124723,5124723] length 1 outage [5124725,5124727] length 3 outage [5124729,5124729] length 1 outage [5124731,5124731] length 1 outage [5124733,5124737] length 5 outage [5124739,5124739] length 1 outage [5124741,5124747] length 7 outage [5124749,5124749] length 1 outage [5124751,5124751] length 1 outage [5124753,5124753] length 1 outage [5124755,5124757] length 3 outage [5124759,5124768] length 10 outage [5124770,5124770] length 1 outage [5124772,5124772] length 1 outage [5124774,5124774] length 1 outage [5124776,5124784] length 9 outage [5124786,5124791] length 6 outage [5124793,5124793] length 1 outage [5124795,5124795] length 1 outage [5124797,5124797] length 1 outage [5124799,5124799] length 1 outage [5124801,5124801] length 1 outage [5124803,5124803] length 1 outage [5124805,5124805] length 1 outage [5124807,5124807] length 1 outage [5124809,5124809] length 1 outage [5124811,5124811] length 1 outage [5124813,5124813] length 1 outage [5124815,5124815] length 1 outage [5124817,5124817] length 1 outage [5124819,5124819] length 1 outage [5124821,5124821] length 1 outage [5124823,5124823] length 1 outage [5124825,5124825] length 1 outage [5124827,5124827] length 1 outage [5124829,5124829] length 1 outage [5124831,5124831] length 1 outage [5124833,5124833] length 1 outage [5124835,5124835] length 1 outage [5124837,5124837] length 1 outage [5124839,5124839] length 1 outage [5124841,5124841] length 1 outage [5124843,5124845] length 3 outage [5124847,5124847] length 1 outage [5124849,5124849] length 1 outage [5124851,5124851] length 1 outage [5124853,5124855] length 3 outage [5124857,5124857] length 1 outage [5124859,5124859] length 1 outage [5124861,5124861] length 1 outage [5124863,5124865] length 3 outage [5124867,5124867] length 1 outage [5124869,5124869] length 1 outage [5124871,5124871] length 1 outage [5124873,5124875] length 3 outage [5124877,5124877] length 1 outage [5124879,5124879] length 1 outage [5124881,5124881] length 1 outage [5124883,5124885] length 3 outage [5124887,5124887] length 1 outage [5124889,5124889] length 1 outage [5124891,5124891] length 1 outage [5124893,5124895] length 3 outage [5124897,5124897] length 1 outage [5124899,5124899] length 1 outage [5124901,5124901] length 1 outage [5124903,5124905] length 3 outage [5124907,5124907] length 1 outage [5124909,5124911] length 3 outage [5124913,5124915] length 3 outage [5124917,5124917] length 1 outage [5124919,5124919] length 1 outage [5124921,5124921] length 1 outage [5124923,5124925] length 3 outage [5124927,5124927] length 1 outage [5124929,5124929] length 1 outage [5124931,5124931] length 1 outage [5124933,5124935] length 3 outage [5124937,5124937] length 1 outage [5124939,5124939] length 1 outage [5124941,5124941] length 1 outage [5124943,5124945] length 3 outage [5124947,5124947] length 1 outage [5124949,5124949] length 1 outage [5124951,5124951] length 1 outage [5124953,5124955] length 3 outage [5124957,5124957] length 1 outage [5124959,5124959] length 1 outage [5124961,5124961] length 1 outage [5124963,5124965] length 3 outage [5124967,5124967] length 1 outage [5124969,5124969] length 1 outage [5124971,5124971] length 1 outage [5124973,5124975] length 3 outage [5124977,5124977] length 1 outage [5124979,5124979] length 1 outage [5124981,5124981] length 1 outage [5124983,5124985] length 3 outage [5124987,5124987] length 1 outage [5124989,5124989] length 1 outage [5124991,5124991] length 1 outage [5124993,5124995] length 3 outage [5124997,5124997] length 1 outage [5124999,5124999] length 1 outage [5125001,5125001] length 1 outage [5125003,5125005] length 3 outage [5125007,5125007] length 1 outage [5125009,5125009] length 1 outage [5125011,5125011] length 1 outage [5125013,5125015] length 3 outage [5125017,5125017] length 1 outage [5125019,5125019] length 1 outage [5125021,5125021] length 1 outage [5125023,5125025] length 3 outage [5125027,5125027] length 1 outage [5125029,5125029] length 1 outage [5125031,5125031] length 1 outage [5125033,5125035] length 3 outage [5125037,5125037] length 1 outage [5125039,5125039] length 1 outage [5125041,5125041] length 1 outage [5125043,5125045] length 3 outage [5125047,5125047] length 1 outage [5125049,5125049] length 1 outage [5125051,5125051] length 1 outage [5125053,5125055] length 3 outage [5125057,5125057] length 1 outage [5125059,5125059] length 1 outage [5125061,5125061] length 1 outage [5125063,5125065] length 3 outage [5125067,5125067] length 1 outage [5125069,5125069] length 1 outage [5125071,5125071] length 1 outage [5125073,5125075] length 3 outage [5125077,5125077] length 1 outage [5125079,5125079] length 1 outage [5125081,5125081] length 1 outage [5125083,5125085] length 3 outage [5125087,5125087] length 1 outage [5125089,5125089] length 1 outage [5125091,5125091] length 1 outage [5125093,5125095] length 3 outage [5125097,5125097] length 1 outage [5125099,5125099] length 1 outage [5125101,5125101] length 1 outage [5125103,5125105] length 3 outage [5125107,5125107] length 1 outage [5125109,5125109] length 1 outage [5125111,5125111] length 1 outage [5125113,5125115] length 3 outage [5125117,5125117] length 1 outage [5125119,5125119] length 1 outage [5125121,5125121] length 1 outage [5125123,5125125] length 3 outage [5125127,5125127] length 1 outage [5125129,5125129] length 1 outage [5125131,5125131] length 1 outage [5125133,5125135] length 3 outage [5125137,5125137] length 1 outage [5125139,5125139] length 1 outage [5125141,5125141] length 1 outage [5125143,5125145] length 3 outage [5125147,5125147] length 1 outage [5125149,5125149] length 1 outage [5125151,5125151] length 1 outage [5125153,5125154] length 2 outage [5125156,5125156] length 1 outage [5125158,5125160] length 3 outage [5125162,5125162] length 1 outage [5125164,5125164] length 1 outage [5125166,5125166] length 1 outage [5125168,5125170] length 3 outage [5125172,5125172] length 1 outage [5125174,5125175] length 2 outage [5125177,5125177] length 1 outage [5125179,5125179] length 1 outage [5125181,5125181] length 1 outage [5125183,5125185] length 3 outage [5125187,5125187] length 1 outage [5125189,5125189] length 1 outage [5125191,5125191] length 1 outage [5125193,5125195] length 3 outage [5125197,5125197] length 1 outage [5125199,5125199] length 1 outage [5125201,5125201] length 1 outage [5125203,5125205] length 3 outage [5125207,5125207] length 1 outage [5125209,5125209] length 1 outage [5125211,5125211] length 1 outage [5125213,5125215] length 3 outage [5125217,5125217] length 1 outage [5125219,5125219] length 1 outage [5125221,5125221] length 1 outage [5125223,5125225] length 3 outage [5125227,5125227] length 1 outage [5125229,5125229] length 1 outage [5125231,5125231] length 1 outage [5125233,5125235] length 3 outage [5125237,5125237] length 1 outage [5125239,5125239] length 1 outage [5125241,5125241] length 1 outage [5125243,5125245] length 3 outage [5125247,5125249] length 3 outage [5125251,5125251] length 1 outage [5125253,5125255] length 3 outage [5125257,5125257] length 1 outage [5125259,5125259] length 1 outage [5125261,5125261] length 1 outage [5125263,5125265] length 3 outage [5125267,5125267] length 1 outage [5125269,5125269] length 1 outage [5125271,5125271] length 1 outage [5125273,5125275] length 3 outage [5125277,5125277] length 1 outage [5125279,5125279] length 1 outage [5125281,5125281] length 1 outage [5125283,5125285] length 3 outage [5125287,5125287] length 1 outage [5125289,5125289] length 1 outage [5125291,5125291] length 1 outage [5125293,5125295] length 3 outage [5125297,5125297] length 1 outage [5125299,5125299] length 1 outage [5125301,5125301] length 1 outage [5125303,5125305] length 3 outage [5125307,5125307] length 1 outage [5125309,5125309] length 1 outage [5125311,5125311] length 1 outage [5125313,5125315] length 3 outage [5125317,5125317] length 1 outage [5125319,5125319] length 1 outage [5125321,5125321] length 1 outage [5125323,5125325] length 3 outage [5125327,5125327] length 1 outage [5125329,5125329] length 1 outage [5125331,5125331] length 1 outage [5125333,5125335] length 3 outage [5125337,5125337] length 1 outage [5125339,5125339] length 1 outage [5125341,5125341] length 1 outage [5125343,5125345] length 3 outage [5125347,5125347] length 1 outage [5125349,5125349] length 1 outage [5125351,5125351] length 1 outage [5125353,5125355] length 3 outage [5125357,5125357] length 1 outage [5125359,5125359] length 1 outage [5125361,5125361] length 1 outage [5125363,5125365] length 3 outage [5125367,5125367] length 1 outage [5125369,5125369] length 1 outage [5125371,5125371] length 1 outage [5125373,5125375] length 3 outage [5125377,5125377] length 1 outage [5125379,5125379] length 1 outage [5125381,5125381] length 1 outage [5125383,5125385] length 3 outage [5125387,5125387] length 1 outage [5125389,5125389] length 1 outage [5125391,5125391] length 1 outage [5125393,5125395] length 3 outage [5125397,5125397] length 1 outage [5125399,5125399] length 1 outage [5125401,5125401] length 1 outage [5125403,5125405] length 3 outage [5125407,5125407] length 1 outage [5125409,5125409] length 1 outage [5125411,5125411] length 1 outage [5125413,5125415] length 3 outage [5125417,5125417] length 1 outage [5125419,5125419] length 1 outage [5125421,5125421] length 1 outage [5125423,5125440] length 18 outage [5125442,5125444] length 3 outage [5125446,5125446] length 1 outage [5125448,5125450] length 3 outage [5125452,5125452] length 1 outage [5125455,5125456] length 2 outage [5125466,5125468] length 3 outage [5125470,5125472] length 3 outage [5125474,5125474] length 1 outage [5125476,5125479] length 4 outage [5125481,5125483] length 3 outage [5125485,5125485] length 1 outage [5125487,5125496] length 10 outage [5125498,5125498] length 1 outage [5125500,5125500] length 1 outage [5125502,5125502] length 1 outage [5125504,5125504] length 1 outage [5125506,5125506] length 1 outage [5125508,5125508] length 1 outage [5125510,5125510] length 1 outage [5125512,5125512] length 1 outage [5125514,5125514] length 1 outage [5125516,5125516] length 1 outage [5125518,5125518] length 1 outage [5125520,5125520] length 1 outage [5125522,5125522] length 1 outage [5125524,5125524] length 1 outage [5125526,5125526] length 1 outage [5125528,5125528] length 1 outage [5125530,5125530] length 1 outage [5125532,5125532] length 1 outage [5125534,5125537] length 4 /dev/label/disk13 /dev/label/disk14 /dev/label/disk15 /dev/label/disk16 raidz /dev/label/disk05 /dev/label/disk06 /dev/label/disk07 /dev/label/disk08 /dev/label/disk17 /dev/label/disk18 /dev/label/disk19 /dev/label/disk20 raidz /dev/label/disk09 /dev/label/disk10 /dev/label/disk11 /dev/label/disk12 /dev/label/disk21 /dev/label/disk22 /dev/label/disk23 /dev/label/disk24
Victor Latushkin
2010-May-01 20:48 UTC
[zfs-discuss] ZFS: "Cannot replace a replacing drive"
On Apr 29, 2010, at 2:20 AM, Freddie Cash wrote:> On Wed, Apr 28, 2010 at 2:48 PM, Victor Latushkin <Victor.Latushkin at sun.com> wrote: > > 2. Run ''zdb -ddd storage'' and provide section titles Dirty Time Logs > > See attached.So you really do have enough redundancy to be able to handle this scenario, so this is software bug. On recent OpenSolaris build you should be able to detach one of the devices, and replace second one. Version 14 corresponds to build 103, and spa_vdev_detach() was changed significantly in build 105 (along with other related changes), so those changes are probably not yet available in FreeBSD.> > 3. Try ''zpool detach'' approach on v14 system? > > Pool upgraded successfully.Btw, you do not have to upgrade pool immediately along with upgrade to newer ZFS version. Now you cannot use upgraded pool on system running older bits.> Same results to all the zpool commands, though: online, offline, detach, replace.Ok, i see. I mistakenly thought that v14 is for user/group quotas which means build 114, but i was wrong - user/group quotas require version 15.> Another option may be to try latest OpenSolaris LiveCD (build 134). > > I''ll have to see if I can download/make one. > > Does it include drivers for 3Ware 9550SXU and 9650SE RAID controllers? All the drives are plugged into those.I do not know for sure, but quick check with google suggests that chances are good. victor -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100502/54472f43/attachment.html>
On Sat, May 1, 2010 at 1:48 PM, Victor Latushkin <Victor.Latushkin at sun.com>wrote:> On Apr 29, 2010, at 2:20 AM, Freddie Cash wrote: > > On Wed, Apr 28, 2010 at 2:48 PM, Victor Latushkin < > Victor.Latushkin at sun.com> wrote: > > >> 2. Run ''zdb -ddd storage'' and provide section titles Dirty Time Logs >> >> See attached. > > So you really do have enough redundancy to be able to handle this scenario, > so this is software bug. On recent OpenSolaris build you should be able to > detach one of the devices, and replace second one. Version 14 corresponds to > build 103, and spa_vdev_detach() was changed significantly in build 105 > (along with other related changes), so those changes are probably not yet > available in FreeBSD. > > After much gnashing of teeth, pulling of hair, reading of web pages, andoffering of dead chickens, the pool is back to an ONLINE state. I pulled the 1.5 TB drive from the system, inserted the original 500 GB drive, and ZFS detected it and marked it as online, with the 1.5 TB as faulted. At that point, I was able to "zpool detach" the 1.5 TB drive, putting the pool back into the original configuration, and online state. After that, I "zpool offline" the 500 GB drive, replace it with a new 1.5 TB drive, and "zpool replace". And it''s happily re-silvering the drive. In 35h, it should be complete. Thankfully, the original drive was not faulty and was only being replaced to increase the size. If the original drive was dead, I''d probably be hooped. -- Freddie Cash fjwcash at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100503/e9d2a6f2/attachment.html>