I''m seeing some odd i/o behaviour on a Sun Fire running snv_70,
connected via 4gb FC to some passthrough disks for a ZFS pool.
The system is normally not heavily loaded, so I don''t pay as much
attention to I/O performance as I should, but recently we had several
drives fail checksums (heat event) and so we''ve been putting ZFS
through it paces on resilvers from spare drives. However, zpool
iostat is being somewhat confusing, as it is showing frequent, longish
periods when no i/o is going on. A very similarly configured box on
the same FC fabric (running snv_72 tagged as of Aug 21) does not
exhibit the timeout behaviour. There are a few scsi timeouts in logs,
but not even remotely enough to account for the ZFS timeouts I''m
seeing.
Really, I''m just looking for ideas on where to start debugging what
might be causing the problem (which results in some really very silly
resilver times).
Config information and sample iostats follow.
Thanks,
Jeff
--
Jeff Bachtel (root at VPR,TAMU) http://www.cepheid.org/~jeff
"The sciences, each straining in [finger jeff at cepheid.org for PGP key]
its own direction, have hitherto harmed us little;" - HPL, TCoC
(Good pool''s zpool status. Error file is a dangling reference to a bad
file that I''ve deleted. Member disks 400gb)
imsfs-mirror:~> sudo zpool status -xv
pool: ims_pool_mirror
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore
the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scrub: scrub in progress, 11.02% done, 15h2m to go
config:
NAME STATE READ WRITE CKSUM
ims_pool_mirror ONLINE 0 0 0
raidz2 ONLINE 0 0 0
c2t21000004D9600099d14 ONLINE 0 0 0
c2t21000004D9600099d1 ONLINE 0 0 0
c2t21000004D9600099d2 ONLINE 0 0 0
c2t21000004D9600099d3 ONLINE 0 0 0
c2t21000004D9600099d15 ONLINE 0 0 0
c2t21000004D9600099d5 ONLINE 0 0 0
c2t21000004D9600099d6 ONLINE 0 0 0
c2t21000004D9600099d7 ONLINE 0 0 0
c2t21000004D9600099d8 ONLINE 0 0 0
c2t21000004D9600099d9 ONLINE 0 0 0
c2t21000004D9600099d10 ONLINE 0 0 0
c2t21000004D9600099d11 ONLINE 0 0 0
c2t21000004D9600099d12 ONLINE 0 0 0
c2t21000004D9600099d13 ONLINE 0 0 0
spares
c2t21000004D9600099d0 AVAIL
c2t21000004D9600099d4 AVAIL
errors: Permanent errors have been detected in the following files:
ims_pool_mirror/backup/vprweb:<0xcad>
(Good pool''s zpool iostat 1 50)
imsfs-mirror:~> sudo zpool iostat 1 50
capacity operations bandwidth
pool used avail read write read write
--------------- ----- ----- ----- ----- ----- -----
ims_pool_mirror 3.84T 1.25T 574 21 65.7M 351K
ims_pool_mirror 3.84T 1.25T 458 0 55.3M 0
ims_pool_mirror 3.84T 1.25T 389 0 47.4M 0
ims_pool_mirror 3.84T 1.25T 532 0 64.2M 0
ims_pool_mirror 3.84T 1.25T 650 0 79.3M 0
ims_pool_mirror 3.84T 1.25T 391 0 47.6M 0
ims_pool_mirror 3.84T 1.25T 548 0 66.2M 0
ims_pool_mirror 3.84T 1.25T 462 0 56.1M 0
ims_pool_mirror 3.84T 1.25T 492 0 59.5M 0
ims_pool_mirror 3.84T 1.25T 488 0 59.6M 0
ims_pool_mirror 3.84T 1.25T 619 0 75.0M 0
ims_pool_mirror 3.84T 1.25T 430 0 52.2M 0
ims_pool_mirror 3.84T 1.25T 467 0 57.1M 0
ims_pool_mirror 3.84T 1.25T 463 0 56.3M 0
ims_pool_mirror 3.84T 1.25T 547 0 66.8M 0
ims_pool_mirror 3.84T 1.25T 513 0 62.2M 0
ims_pool_mirror 3.84T 1.25T 449 0 54.5M 0
ims_pool_mirror 3.84T 1.25T 445 0 53.6M 0
ims_pool_mirror 3.84T 1.25T 501 0 61.4M 0
ims_pool_mirror 3.84T 1.25T 558 0 68.1M 0
ims_pool_mirror 3.84T 1.25T 718 0 87.5M 0
ims_pool_mirror 3.84T 1.25T 385 0 47.0M 0
ims_pool_mirror 3.84T 1.25T 415 0 50.2M 0
ims_pool_mirror 3.84T 1.25T 626 0 76.1M 0
ims_pool_mirror 3.84T 1.25T 579 0 70.6M 0
ims_pool_mirror 3.84T 1.25T 516 0 62.9M 0
ims_pool_mirror 3.84T 1.25T 465 0 56.5M 0
ims_pool_mirror 3.84T 1.25T 601 0 73.2M 0
ims_pool_mirror 3.84T 1.25T 361 0 44.5M 0
ims_pool_mirror 3.84T 1.25T 335 0 40.0M 0
ims_pool_mirror 3.84T 1.25T 473 0 57.5M 0
ims_pool_mirror 3.84T 1.25T 432 0 52.8M 0
ims_pool_mirror 3.84T 1.25T 668 0 81.7M 0
ims_pool_mirror 3.84T 1.25T 607 0 74.2M 0
ims_pool_mirror 3.84T 1.25T 557 0 67.4M 0
ims_pool_mirror 3.84T 1.25T 425 0 51.5M 0
ims_pool_mirror 3.84T 1.25T 450 0 55.0M 0
ims_pool_mirror 3.84T 1.25T 688 0 83.6M 0
ims_pool_mirror 3.84T 1.25T 524 0 63.8M 0
ims_pool_mirror 3.84T 1.25T 798 0 96.8M 0
ims_pool_mirror 3.84T 1.25T 343 0 41.9M 0
ims_pool_mirror 3.84T 1.25T 583 0 71.2M 0
ims_pool_mirror 3.84T 1.25T 319 0 38.9M 0
ims_pool_mirror 3.84T 1.25T 571 0 69.0M 0
ims_pool_mirror 3.84T 1.25T 461 0 56.4M 0
ims_pool_mirror 3.84T 1.25T 445 0 54.0M 0
ims_pool_mirror 3.84T 1.25T 428 0 52.3M 0
ims_pool_mirror 3.84T 1.25T 569 0 68.8M 0
ims_pool_mirror 3.84T 1.25T 348 0 42.4M 0
ims_pool_mirror 3.84T 1.25T 563 0 68.7M 0
(Slow pool''s zpool status. Member disks 750gb)
imsfs:~> sudo zpool status -xv
pool: ims_pool
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore
the
entire pool from backup.
see: http://www.sun.com/msg/ZFS-8000-8A
scrub: resilver in progress, 0.72% done, 51h34m to go
config:
NAME STATE READ WRITE CKSUM
ims_pool DEGRADED 0 0 8
raidz2 DEGRADED 0 0 8
spare DEGRADED 0 0 0
c3t21000004D960CDEDd0 DEGRADED 0 0 0 too many
errors
c3t21000004D960CDEDd7 ONLINE 0 0 0
spare DEGRADED 0 0 0
c3t21000004D960CDEDd1 DEGRADED 0 0 0 too many
errors
c3t21000004D960CDEDd9 ONLINE 0 0 0
c3t21000004D960CDEDd2 ONLINE 0 0 0
spare DEGRADED 0 0 0
c3t21000004D960CDEDd3 DEGRADED 0 0 0 too many
errors
c3t21000004D960CDEDd8 ONLINE 0 0 0
spare DEGRADED 0 0 0
c3t21000004D960CDEDd4 DEGRADED 0 0 0 too many
errors
c3t21000004D960CDEDd10 ONLINE 0 0 0
spare DEGRADED 0 0 0
c3t21000004D960CDEDd5 DEGRADED 0 0 0 too many
errors
c3t21000004D960CDEDd11 ONLINE 0 0 0
c3t21000004D960CDEDd6 ONLINE 0 0 0
spares
c3t21000004D960CDEDd7 INUSE currently in use
c3t21000004D960CDEDd8 INUSE currently in use
c3t21000004D960CDEDd9 INUSE currently in use
c3t21000004D960CDEDd10 INUSE currently in use
c3t21000004D960CDEDd11 INUSE currently in use
errors: Permanent errors have been detected in the following files:
/export/ims/content/Archive/temp_8QLgIp8a2xHwV-kOq3THBQ==.tmp
/export/ims/content/test_h.mov
(Slow pool zpool iostat 1 50)
imsfs:~> sudo zpool iostat 1 50
capacity operations bandwidth
pool used avail read write read write
---------- ----- ----- ----- ----- ----- -----
ims_pool 3.81T 958G 129 18 14.5M 50.1K
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 8 0 35.6K
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 3 0 507K 0
ims_pool 3.81T 958G 0 0 0 0
ims_pool 3.81T 958G 526 139 64.1M 439K
ims_pool 3.81T 958G 915 5 113M 34.3K
ims_pool 3.81T 958G 996 9 122M 51.1K
ims_pool 3.81T 958G 709 4 87.2M 26.3K
ims_pool 3.81T 958G 90 3 11.0M 19.3K
ims_pool 3.81T 958G 983 4 121M 27.0K
ims_pool 3.81T 958G 914 11 112M 63.6K
ims_pool 3.81T 958G 1.09K 5 138M 34.1K
ims_pool 3.81T 958G 964 11 118M 57.2K