Ok, this is not a OpenSolaris question, but it is a Solaris and ZFS
question.
I have a pool with three mirrored vdevs. I just got an error message
from FMD that read failed from one on the disks,(c1t6d0). All with
instructions on how to handle the problem and replace the devices, so
far everything is good. But the zpool still thinks everything is fine.
Shouldn''t zpool also show errors in this state?
This was run on S10U4 with 127127-11.
# zpool status -x
all pools are healthy
# zpool status
pool: storage
state: ONLINE
scrub: scrub completed with 0 errors on Sun Jun 29 23:16:34 2008
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
c1t4d0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c1t5d0 ONLINE 0 0 0
c1t6d0 ONLINE 0 0 0
errors: No known data errors
# fmdump -v
TIME UUID SUNW-MSG-ID
Jul 08 20:14:42.6951 3780a675-96ea-6fa4-bd55-cb078a539f08 ZFS-8000-D3
100% fault.fs.zfs.device
Problem in: zfs://pool=storage/vdev=83de319aad25c131
Affects: zfs://pool=storage/vdev=83de319aad25c131
FRU: -
Location: -
From my message log:
Jul 8 20:11:53 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:11:53 fortress SCSI transport failed: reason
''incomplete'': retrying command
Jul 8 20:12:56 fortress scsi: [ID 365881 kern.info] fas: 6.0:
cdb=[ 0xa 0x0 0x1 0xda 0x2 0x0 ]
Jul 8 20:12:56 fortress scsi: [ID 365881 kern.info] fas: 6.0:
cdb=[ 0xa 0x0 0x3 0xda 0x2 0x0 ]
Jul 8 20:12:56 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000 (fas1):
Jul 8 20:12:56 fortress Disconnected tagged cmd(s) (2) timeout
for Target 6.0
Jul 8 20:12:56 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:12:56 fortress SCSI transport failed: reason
''timeout'': retrying command
Jul 8 20:12:56 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:12:56 fortress SCSI transport failed: reason
''reset'':
retrying command
Jul 8 20:12:59 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:12:59 fortress Error for Command:
write(10) Error Level: Retryable
Jul 8 20:12:59 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17672154 Error Block: 17672154
Jul 8 20:12:59 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:12:59 fortress scsi: [ID 107833 kern.notice] Sense Key:
Unit Attention
Jul 8 20:12:59 fortress scsi: [ID 107833 kern.notice] ASC: 0x29
(power on occurred), ASCQ: 0x1, FRU: 0x1
Jul 8 20:12:59 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:12:59 fortress Error for Command:
write(10) Error Level: Retryable
Jul 8 20:12:59 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17672154 Error Block: 17672154
Jul 8 20:12:59 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:12:59 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:12:59 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:04 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:04 fortress Error for Command:
write(10) Error Level: Retryable
Jul 8 20:13:04 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17672154 Error Block: 17672154
Jul 8 20:13:04 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:04 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:04 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:09 fortress Error for Command:
write(10) Error Level: Fatal
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17672154 Error Block: 17672154
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:09 fortress Error for Command:
write(10) Error Level: Retryable
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17672666 Error Block: 17672666
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:09 fortress Error for Command:
write Error Level: Retryable
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Requested
Block: 986 Error Block: 986
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:09 fortress Error for Command:
write Error Level: Retryable
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Requested
Block: 474 Error Block: 474
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:09 fortress Error for Command:
read Error Level: Retryable
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Requested
Block: 546 Error Block: 546
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:09 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:14 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:14 fortress Error for Command:
write(10) Error Level: Retryable
Jul 8 20:13:14 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17672666 Error Block: 17672666
Jul 8 20:13:14 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:14 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:14 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:19 fortress Error for Command:
write(10) Error Level: Fatal
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17672666 Error Block: 17672666
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:19 fortress Error for Command:
read Error Level: Retryable
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Requested
Block: 546 Error Block: 546
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:19 fortress Error for Command:
write Error Level: Retryable
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Requested
Block: 474 Error Block: 474
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:19 fortress Error for Command:
write Error Level: Retryable
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Requested
Block: 986 Error Block: 986
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:19 fortress Error for Command:
read Error Level: Retryable
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Requested
Block: 546 Error Block: 546
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:19 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:24 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:24 fortress Error for Command:
read Error Level: Fatal
Jul 8 20:13:24 fortress scsi: [ID 107833 kern.notice] Requested
Block: 546 Error Block: 546
Jul 8 20:13:24 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:24 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:24 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:41 fortress Error for Command:
read Error Level: Retryable
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.notice] Requested
Block: 546 Error Block: 546
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:41 fortress Error for Command:
write Error Level: Fatal
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.notice] Requested
Block: 986 Error Block: 986
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:41 fortress Error for Command:
write Error Level: Fatal
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.notice] Requested
Block: 474 Error Block: 474
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:41 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:46 fortress Error for Command:
read Error Level: Fatal
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.notice] Requested
Block: 546 Error Block: 546
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:46 fortress Error for Command:
read(10) Error Level: Retryable
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17672226 Error Block: 17672226
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:46 fortress Error for Command:
read(10) Error Level: Retryable
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17671714 Error Block: 17671714
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:46 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:51 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:51 fortress Error for Command:
read(10) Error Level: Retryable
Jul 8 20:13:51 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17672226 Error Block: 17672226
Jul 8 20:13:51 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:51 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:51 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:56 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:56 fortress Error for Command:
read(10) Error Level: Fatal
Jul 8 20:13:56 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17672226 Error Block: 17672226
Jul 8 20:13:56 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:56 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:56 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:13:56 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:13:56 fortress Error for Command:
read(10) Error Level: Retryable
Jul 8 20:13:56 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17671714 Error Block: 17671714
Jul 8 20:13:56 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:13:56 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:13:56 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:14:01 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:14:01 fortress Error for Command:
read(10) Error Level: Fatal
Jul 8 20:14:01 fortress scsi: [ID 107833 kern.notice] Requested
Block: 17671714 Error Block: 17671714
Jul 8 20:14:01 fortress scsi: [ID 107833 kern.notice] Vendor:
SEAGATE Serial Number: 9946626576
Jul 8 20:14:01 fortress scsi: [ID 107833 kern.notice] Sense Key: Not
Ready
Jul 8 20:14:01 fortress scsi: [ID 107833 kern.notice] ASC: 0x4 (LUN
is becoming ready), ASCQ: 0x1, FRU: 0x2
Jul 8 20:14:22 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:14:22 fortress offline or reservation conflict
Jul 8 20:14:42 fortress scsi: [ID 107833 kern.warning] WARNING: /
sbus at 1f,0/SUNW,fas at 2,8800000/sd at 6,0 (sd19):
Jul 8 20:14:42 fortress offline or reservation conflict
Jul 8 20:14:42 fortress fmd: [ID 441519 daemon.error] SUNW-MSG-ID:
ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major
Jul 8 20:14:42 fortress EVENT-TIME: Tue Jul 8 20:14:42 CEST 2008
Jul 8 20:14:42 fortress PLATFORM: SUNW,Ultra-2, CSN: -, HOSTNAME:
fortress
Jul 8 20:14:42 fortress SOURCE: zfs-diagnosis, REV: 1.0
Jul 8 20:14:42 fortress EVENT-ID: 3780a675-96ea-6fa4-bd55-cb078a539f08
Jul 8 20:14:42 fortress DESC: A ZFS device failed. Refer to
http://sun.com/msg/ZFS-8000-D3
for more information.
Jul 8 20:14:42 fortress AUTO-RESPONSE: No automated response will
occur.
Jul 8 20:14:42 fortress IMPACT: Fault tolerance of the pool may be
compromised.
Jul 8 20:14:42 fortress REC-ACTION: Run ''zpool status -x'' and
replace
the bad device.
