Just look at this. I thought all the restarting resilver bugs were fixed, but it looks like something odd is still happening at the start: Status immediately after starting resilver: # zpool status pool: rc-pool state: DEGRADED status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using ''zpool clear'' or replace the device with ''zpool replace''. see: http://www.sun.com/msg/ZFS-8000-9P scrub: resilver in progress for 0h0m, 0.00% done, 57h3m to go config: NAME STATE READ WRITE CKSUM rc-pool DEGRADED 0 0 0 mirror DEGRADED 0 0 0 c4t1d0 ONLINE 0 0 0 5.56M resilvered replacing DEGRADED 0 0 0 c4t2d0s0/o FAULTED 1.71M 23.3M 0 too many errors c4t2d0 ONLINE 0 0 0 5.43M resilvered c5t1d0 ONLINE 0 0 0 5.55M resilvered And a few minutes later: # zpool status pool: rc-pool state: DEGRADED status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using ''zpool clear'' or replace the device with ''zpool replace''. see: http://www.sun.com/msg/ZFS-8000-9P scrub: resilver in progress for 0h0m, 0.00% done, 245h21m to go config: NAME STATE READ WRITE CKSUM rc-pool DEGRADED 0 0 0 mirror DEGRADED 0 0 0 c4t1d0 ONLINE 0 0 0 1.10M resilvered replacing DEGRADED 0 0 0 c4t2d0s0/o FAULTED 1.71M 23.3M 0 too many errors c4t2d0 ONLINE 0 0 0 824K resilvered c5t1d0 ONLINE 0 0 0 1.10M resilvered It''s gone from 5MB resilvered to 1MB, and increased the estimated time to 245 hours. -- This message posted from opensolaris.org
Ross, I feel you here, but I don''t have much of a solution. The best I can suggest (and has been my solution) is to take out the problematic disk, copy it to a fresh disk (preferably using something like dd_rescue) and then re-install. It seems the resilvering loop is generally a result of a faulty device, but even if it is taken offline, you still have issues. I have had so many zpool resilvering loops, it''s not funny. I''m running 2009.06 with all updates applied. I''ve had a very, very bad batch of disks. I actually have a resilvering loop running right now, and I need to go copy off the offending device. Again. I wish I had a better solution, because the zpool functions fine, no data errors, but resilvering loops forever. I love ZFS as an on-disk format. I increasingly hate the implementation of ZFS software. -Galen On Jul 13, 2009, at 5:34 AM, Ross wrote:> Just look at this. I thought all the restarting resilver bugs were > fixed, but it looks like something odd is still happening at the > start: > > Status immediately after starting resilver: > > # zpool status > pool: rc-pool > state: DEGRADED > status: One or more devices has experienced an unrecoverable error. > An > attempt was made to correct the error. Applications are > unaffected. > action: Determine if the device needs to be replaced, and clear the > errors > using ''zpool clear'' or replace the device with ''zpool replace''. > see: http://www.sun.com/msg/ZFS-8000-9P > scrub: resilver in progress for 0h0m, 0.00% done, 57h3m to go > config: > > NAME STATE READ WRITE CKSUM > rc-pool DEGRADED 0 0 0 > mirror DEGRADED 0 0 0 > c4t1d0 ONLINE 0 0 0 5.56M resilvered > replacing DEGRADED 0 0 0 > c4t2d0s0/o FAULTED 1.71M 23.3M 0 too many errors > c4t2d0 ONLINE 0 0 0 5.43M resilvered > c5t1d0 ONLINE 0 0 0 5.55M resilvered > > > And a few minutes later: > > # zpool status > pool: rc-pool > state: DEGRADED > status: One or more devices has experienced an unrecoverable error. > An > attempt was made to correct the error. Applications are > unaffected. > action: Determine if the device needs to be replaced, and clear the > errors > using ''zpool clear'' or replace the device with ''zpool replace''. > see: http://www.sun.com/msg/ZFS-8000-9P > scrub: resilver in progress for 0h0m, 0.00% done, 245h21m to go > config: > > NAME STATE READ WRITE CKSUM > rc-pool DEGRADED 0 0 0 > mirror DEGRADED 0 0 0 > c4t1d0 ONLINE 0 0 0 1.10M resilvered > replacing DEGRADED 0 0 0 > c4t2d0s0/o FAULTED 1.71M 23.3M 0 too many errors > c4t2d0 ONLINE 0 0 0 824K resilvered > c5t1d0 ONLINE 0 0 0 1.10M resilvered > > > It''s gone from 5MB resilvered to 1MB, and increased the estimated > time to 245 hours. > -- > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Ross Walker
2009-Jul-13 13:50 UTC
[zfs-discuss] OpenSolaris 2008.11 - resilver still restarting
Maybe it''s the disks firmware that is bad or maybe they''re jumpered for 1.5Gbps on a 3.0 only bus? Or maybe it''s a problem with the disk cable/bay/enclosure/slot? It sounds like there is more then ZFS in the mix here. I wonder if the drive''s status keeps flapping online/offline and either ZFS or FMA are too lax in marking a drive offline after recurring timeouts. Take a look at your disk enclosure and iostat -En for the number of timeouts happenning. -Ross On Jul 13, 2009, at 9:05 AM, Galen <galenz at zinkconsulting.com> wrote:> Ross, > > I feel you here, but I don''t have much of a solution. > > The best I can suggest (and has been my solution) is to take out the > problematic disk, copy it to a fresh disk (preferably using > something like dd_rescue) and then re-install. > > It seems the resilvering loop is generally a result of a faulty > device, but even if it is taken offline, you still have issues. I > have had so many zpool resilvering loops, it''s not funny. I''m > running 2009.06 with all updates applied. I''ve had a very, very bad > batch of disks. > > I actually have a resilvering loop running right now, and I need to > go copy off the offending device. Again. > > I wish I had a better solution, because the zpool functions fine, no > data errors, but resilvering loops forever. I love ZFS as an on-disk > format. I increasingly hate the implementation of ZFS software. > > -Galen > > On Jul 13, 2009, at 5:34 AM, Ross wrote: > >> Just look at this. I thought all the restarting resilver bugs were >> fixed, but it looks like something odd is still happening at the >> start: >> >> Status immediately after starting resilver: >> >> # zpool status >> pool: rc-pool >> state: DEGRADED >> status: One or more devices has experienced an unrecoverable >> error. An >> attempt was made to correct the error. Applications are >> unaffected. >> action: Determine if the device needs to be replaced, and clear the >> errors >> using ''zpool clear'' or replace the device with ''zpool replace''. >> see: http://www.sun.com/msg/ZFS-8000-9P >> scrub: resilver in progress for 0h0m, 0.00% done, 57h3m to go >> config: >> >> NAME STATE READ WRITE CKSUM >> rc-pool DEGRADED 0 0 0 >> mirror DEGRADED 0 0 0 >> c4t1d0 ONLINE 0 0 0 5.56M resilvered >> replacing DEGRADED 0 0 0 >> c4t2d0s0/o FAULTED 1.71M 23.3M 0 too many errors >> c4t2d0 ONLINE 0 0 0 5.43M resilvered >> c5t1d0 ONLINE 0 0 0 5.55M resilvered >> >> >> And a few minutes later: >> >> # zpool status >> pool: rc-pool >> state: DEGRADED >> status: One or more devices has experienced an unrecoverable >> error. An >> attempt was made to correct the error. Applications are >> unaffected. >> action: Determine if the device needs to be replaced, and clear the >> errors >> using ''zpool clear'' or replace the device with ''zpool replace''. >> see: http://www.sun.com/msg/ZFS-8000-9P >> scrub: resilver in progress for 0h0m, 0.00% done, 245h21m to go >> config: >> >> NAME STATE READ WRITE CKSUM >> rc-pool DEGRADED 0 0 0 >> mirror DEGRADED 0 0 0 >> c4t1d0 ONLINE 0 0 0 1.10M resilvered >> replacing DEGRADED 0 0 0 >> c4t2d0s0/o FAULTED 1.71M 23.3M 0 too many errors >> c4t2d0 ONLINE 0 0 0 824K resilvered >> c5t1d0 ONLINE 0 0 0 1.10M resilvered >> >> >> It''s gone from 5MB resilvered to 1MB, and increased the estimated >> time to 245 hours. >> -- >> This message posted from opensolaris.org >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
No, I don''t think I need to take a disk out. It''s running ok now, it just seemed to get a bit confused at the start: $ zpool status pool: rc-pool state: DEGRADED status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using ''zpool clear'' or replace the device with ''zpool replace''. see: http://www.sun.com/msg/ZFS-8000-9P scrub: resilver in progress for 2h11m, 31.09% done, 4h51m to go config: NAME STATE READ WRITE CKSUM rc-pool DEGRADED 0 0 0 mirror DEGRADED 0 0 0 c4t1d0 ONLINE 0 0 0 101M resilvered replacing DEGRADED 0 0 0 c4t2d0s0/o FAULTED 1.71M 23.3M 0 too many errors c4t2d0 ONLINE 0 0 0 62.3G resilvered c5t1d0 ONLINE 0 0 0 101M resilvered mirror ONLINE 0 0 0 c4t3d0 ONLINE 0 0 0 c5t2d0 ONLINE 0 0 0 c5t0d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t3d0 ONLINE 0 0 0 c4t5d0 ONLINE 0 0 0 c4t4d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t4d0 ONLINE 0 0 0 c5t5d0 ONLINE 0 0 0 c4t6d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c4t7d0 ONLINE 0 13.0K 0 c5t6d0 ONLINE 0 0 0 c5t7d0 ONLINE 0 0 0 logs DEGRADED 0 0 0 c6d1p0 ONLINE 0 0 0 errors: No known data errors -- This message posted from opensolaris.org
Gaaah, looks like I spoke too soon: $ zpool status pool: rc-pool state: DEGRADED status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using ''zpool clear'' or replace the device with ''zpool replace''. see: http://www.sun.com/msg/ZFS-8000-9P scrub: resilver in progress for 2h59m, 77.89% done, 0h50m to go config: NAME STATE READ WRITE CKSUM rc-pool DEGRADED 0 0 0 mirror DEGRADED 0 0 0 c4t1d0 ONLINE 0 0 0 218M resilvered replacing UNAVAIL 0 963K 0 insufficient replicas c4t2d0s0/o FAULTED 1.71M 23.4M 0 too many errors c4t2d0 REMOVED 0 964K 0 67.0G resilvered c5t1d0 ONLINE 0 0 0 218M resilvered mirror ONLINE 0 0 0 c4t3d0 ONLINE 0 0 0 c5t2d0 ONLINE 0 0 0 c5t0d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t3d0 ONLINE 0 0 0 c4t5d0 ONLINE 0 0 0 c4t4d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t4d0 ONLINE 0 0 0 c5t5d0 ONLINE 0 0 0 c4t6d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c4t7d0 ONLINE 0 13.0K 0 c5t6d0 ONLINE 0 0 0 c5t7d0 ONLINE 0 0 0 logs DEGRADED 0 0 0 c6d1p0 ONLINE 0 0 0 errors: No known data errors There are a whole bunch of errors in /var/adm/messages: Jul 13 15:56:53 rob-036 scsi: [ID 107833 kern.warning] WARNING: /pci at 1,0/pci1022,7458 at 1/pci11ab,11ab at 2/disk at 2,0 (sd3): Jul 13 15:56:53 rob-036 Error for Command: write(10) Error Level: Retryable Jul 13 15:56:53 rob-036 scsi: [ID 107833 kern.notice] Requested Block: 83778048 Error Block: 83778048 Jul 13 15:56:53 rob-036 scsi: [ID 107833 kern.notice] Vendor: ATA Serial Number: Jul 13 15:56:53 rob-036 scsi: [ID 107833 kern.notice] Sense Key: Aborted_Command Jul 13 15:56:53 rob-036 scsi: [ID 107833 kern.notice] ASC: 0x0 (no additional sense info), ASCQ: 0x0, FRU: 0x0 Jul 13 15:57:31 rob-036 scsi: [ID 107833 kern.warning] WARNING: /pci at 1,0/pci1022,7458 at 1/pci11ab,11ab at 2/disk at 2,0 (sd3): Jul 13 15:57:31 rob-036 Command failed to complete...Device is gone Not what I would expect from a brand new drive!! Does anybody have any tips on how i can work out where the fault lies here? I wouldn''t expect controller with so many other drives working, and what on earth is the proper technique for replacing a drive that failed part way through a resilver? -- This message posted from opensolaris.org
Ross, The disks do have problems - that''s why I''m resilvering. I''ve seen zero read, write or checksum errors and had it loop. Now I do have a number of read errors on some of the disks, but I think resilvering is missing the point if it can''t deal with corrupt data or disks with a small amount of unreadable data. -Galen On Jul 13, 2009, at 6:50 AM, Ross Walker wrote:> > Maybe it''s the disks firmware that is bad or maybe they''re jumpered > for 1.5Gbps on a 3.0 only bus? Or maybe it''s a problem with the disk > cable/bay/enclosure/slot? > > It sounds like there is more then ZFS in the mix here. I wonder if > the drive''s status keeps flapping online/offline and either ZFS or > FMA are too lax in marking a drive offline after recurring timeouts. > > Take a look at your disk enclosure and iostat -En for the number of > timeouts happenning. > > -Ross > > > On Jul 13, 2009, at 9:05 AM, Galen <galenz at zinkconsulting.com> wrote: > >> Ross, >> >> I feel you here, but I don''t have much of a solution. >> >> The best I can suggest (and has been my solution) is to take out >> the problematic disk, copy it to a fresh disk (preferably using >> something like dd_rescue) and then re-install. >> >> It seems the resilvering loop is generally a result of a faulty >> device, but even if it is taken offline, you still have issues. I >> have had so many zpool resilvering loops, it''s not funny. I''m >> running 2009.06 with all updates applied. I''ve had a very, very bad >> batch of disks. >> >> I actually have a resilvering loop running right now, and I need to >> go copy off the offending device. Again. >> >> I wish I had a better solution, because the zpool functions fine, >> no data errors, but resilvering loops forever. I love ZFS as an on- >> disk format. I increasingly hate the implementation of ZFS software. >> >> -Galen >> >> On Jul 13, 2009, at 5:34 AM, Ross wrote: >> >>> Just look at this. I thought all the restarting resilver bugs >>> were fixed, but it looks like something odd is still happening at >>> the start: >>> >>> Status immediately after starting resilver: >>> >>> # zpool status >>> pool: rc-pool >>> state: DEGRADED >>> status: One or more devices has experienced an unrecoverable >>> error. An >>> attempt was made to correct the error. Applications are >>> unaffected. >>> action: Determine if the device needs to be replaced, and clear >>> the errors >>> using ''zpool clear'' or replace the device with ''zpool replace''. >>> see: http://www.sun.com/msg/ZFS-8000-9P >>> scrub: resilver in progress for 0h0m, 0.00% done, 57h3m to go >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> rc-pool DEGRADED 0 0 0 >>> mirror DEGRADED 0 0 0 >>> c4t1d0 ONLINE 0 0 0 5.56M resilvered >>> replacing DEGRADED 0 0 0 >>> c4t2d0s0/o FAULTED 1.71M 23.3M 0 too many errors >>> c4t2d0 ONLINE 0 0 0 5.43M resilvered >>> c5t1d0 ONLINE 0 0 0 5.55M resilvered >>> >>> >>> And a few minutes later: >>> >>> # zpool status >>> pool: rc-pool >>> state: DEGRADED >>> status: One or more devices has experienced an unrecoverable >>> error. An >>> attempt was made to correct the error. Applications are >>> unaffected. >>> action: Determine if the device needs to be replaced, and clear >>> the errors >>> using ''zpool clear'' or replace the device with ''zpool replace''. >>> see: http://www.sun.com/msg/ZFS-8000-9P >>> scrub: resilver in progress for 0h0m, 0.00% done, 245h21m to go >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> rc-pool DEGRADED 0 0 0 >>> mirror DEGRADED 0 0 0 >>> c4t1d0 ONLINE 0 0 0 1.10M resilvered >>> replacing DEGRADED 0 0 0 >>> c4t2d0s0/o FAULTED 1.71M 23.3M 0 too many errors >>> c4t2d0 ONLINE 0 0 0 824K resilvered >>> c5t1d0 ONLINE 0 0 0 1.10M resilvered >>> >>> >>> It''s gone from 5MB resilvered to 1MB, and increased the estimated >>> time to 245 hours. >>> -- >>> This message posted from opensolaris.org >>> _______________________________________________ >>> zfs-discuss mailing list >>> zfs-discuss at opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Ross Walker
2009-Jul-13 15:51 UTC
[zfs-discuss] OpenSolaris 2008.11 - resilver still restarting
On Jul 13, 2009, at 11:33 AM, Ross <no-reply at opensolaris.org> wrote:> Gaaah, looks like I spoke too soon: > > $ zpool status > pool: rc-pool > state: DEGRADED > status: One or more devices has experienced an unrecoverable error. > An > attempt was made to correct the error. Applications are > unaffected. > action: Determine if the device needs to be replaced, and clear the > errors > using ''zpool clear'' or replace the device with ''zpool replace''. > see: http://www.sun.com/msg/ZFS-8000-9P > scrub: resilver in progress for 2h59m, 77.89% done, 0h50m to go > config: > > NAME STATE READ WRITE CKSUM > rc-pool DEGRADED 0 0 0 > mirror DEGRADED 0 0 0 > c4t1d0 ONLINE 0 0 0 218M resilvered > replacing UNAVAIL 0 963K 0 insufficient > replicas > c4t2d0s0/o FAULTED 1.71M 23.4M 0 too many errors > c4t2d0 REMOVED 0 964K 0 67.0G resilvered > c5t1d0 ONLINE 0 0 0 218M resilvered > mirror ONLINE 0 0 0 > c4t3d0 ONLINE 0 0 0 > c5t2d0 ONLINE 0 0 0 > c5t0d0 ONLINE 0 0 0 > mirror ONLINE 0 0 0 > c5t3d0 ONLINE 0 0 0 > c4t5d0 ONLINE 0 0 0 > c4t4d0 ONLINE 0 0 0 > mirror ONLINE 0 0 0 > c5t4d0 ONLINE 0 0 0 > c5t5d0 ONLINE 0 0 0 > c4t6d0 ONLINE 0 0 0 > mirror ONLINE 0 0 0 > c4t7d0 ONLINE 0 13.0K 0 > c5t6d0 ONLINE 0 0 0 > c5t7d0 ONLINE 0 0 0 > logs DEGRADED 0 0 0 > c6d1p0 ONLINE 0 0 0 > > errors: No known data errors > > > There are a whole bunch of errors in /var/adm/messages: > > Jul 13 15:56:53 rob-036 scsi: [ID 107833 kern.warning] WARNING: / > pci at 1,0/pci1022,7458 at 1/pci11ab,11ab at 2/disk at 2,0 (sd3): > Jul 13 15:56:53 rob-036 Error for Command: write > (10) Error Level: Retryable > Jul 13 15:56:53 rob-036 scsi: [ID 107833 kern.notice] Requested > Block: 83778048 Error Block: 83778048 > Jul 13 15:56:53 rob-036 scsi: [ID 107833 kern.notice] Vendor: > ATA Serial Number: > Jul 13 15:56:53 rob-036 scsi: [ID 107833 kern.notice] Sense Key: > Aborted_Command > Jul 13 15:56:53 rob-036 scsi: [ID 107833 kern.notice] ASC: 0x0 (no > additional sense info), ASCQ: 0x0, FRU: 0x0 > > > Jul 13 15:57:31 rob-036 scsi: [ID 107833 kern.warning] WARNING: / > pci at 1,0/pci1022,7458 at 1/pci11ab,11ab at 2/disk at 2,0 (sd3): > Jul 13 15:57:31 rob-036 Command failed to complete...Device > is gone > > > Not what I would expect from a brand new drive!! > > Does anybody have any tips on how i can work out where the fault > lies here? I wouldn''t expect controller with so many other drives > working, and what on earth is the proper technique for replacing a > drive that failed part way through a resilver?I really believe there is a problem with either the cabling or the enclosure''s backplane here. Two disks is statistical coincidence, three disks means, it ain''t the disks that are bad (if you checked and there was no recall and the firmware is correct and up to date). Fix the real problem and the disks already in place should resilver without further interruption. -Ross
Rather bizarrely, after that second failure I pulled the disk, cleared the pool, re-inserted it and forced it online. This time, ZFS resilvered fine with zero errors: # zpool status pool: rc-pool state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using ''zpool upgrade''. Once this is done, the pool will no longer be accessible on older software versions. scrub: resilver completed after 4h31m with 0 errors on Mon Jul 13 21:24:51 2009 config: NAME STATE READ WRITE CKSUM rc-pool ONLINE 0 0 0 mirror ONLINE 0 0 0 c4t1d0 ONLINE 0 0 0 195M resilvered c4t2d0 ONLINE 0 0 0 134G resilvered c5t1d0 ONLINE 0 0 0 195M resilvered mirror ONLINE 0 0 0 c4t3d0 ONLINE 0 0 0 c5t2d0 ONLINE 0 0 0 c5t0d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t3d0 ONLINE 0 0 0 c4t5d0 ONLINE 0 0 0 c4t4d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t4d0 ONLINE 0 0 0 c5t5d0 ONLINE 0 0 0 c4t6d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c4t7d0 ONLINE 0 0 0 c5t6d0 ONLINE 0 0 0 c5t7d0 ONLINE 0 0 0 logs ONLINE 0 0 0 c6d1p0 ONLINE 0 0 0 After the resilver I''ve run a scrub, and now the whole pool reports a clean bill of health. # zpool status pool: rc-pool state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using ''zpool upgrade''. Once this is done, the pool will no longer be accessible on older software versions. scrub: scrub completed after 3h50m with 0 errors on Tue Jul 14 12:03:03 2009 config: NAME STATE READ WRITE CKSUM rc-pool ONLINE 0 0 0 mirror ONLINE 0 0 0 c4t1d0 ONLINE 0 0 0 c4t2d0 ONLINE 0 0 0 c5t1d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c4t3d0 ONLINE 0 0 0 c5t2d0 ONLINE 0 0 0 c5t0d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t3d0 ONLINE 0 0 0 c4t5d0 ONLINE 0 0 0 c4t4d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c5t4d0 ONLINE 0 0 0 c5t5d0 ONLINE 0 0 0 c4t6d0 ONLINE 0 0 0 mirror ONLINE 0 0 0 c4t7d0 ONLINE 0 0 0 c5t6d0 ONLINE 0 0 0 c5t7d0 ONLINE 0 0 0 logs ONLINE 0 0 0 c6d1p0 ONLINE 0 0 0 errors: No known data errors I don''t know whether this was just a bad connection, but I''ve now had ZFS do exactly the same on two machines. On both of them, I''ve replaced a disk with a brand new one and had it complain of errors with the replacement. And again, on both machines after clearing the errors and starting the rebuild again it completed fine. I''m going to test both of these disks carefully, I find it very odd for ZFS to complain of errors when replacing disks on two separate machines. -- This message posted from opensolaris.org