Hi, I am looking for the ideas on how to check if the machine was under high I/O pressure before it panicked (caused manually by an NMI). By I/O I mean disks and ZFS stack. -- Piotr Jasiukajtis | estibi | SCA OS0072 http://estseg.blogspot.com
Since on-discuss is no longer maintained I''m adding illumos-developer to this thread. On Mon, Sep 6, 2010 at 11:56 AM, Piotr Jasiukajtis <estseg at gmail.com> wrote:> Hi, > > I am looking for the ideas on how to check if the machine was under > high I/O pressure before it panicked (caused manually by an NMI). > By I/O I mean disks and ZFS stack.-- Piotr Jasiukajtis | estibi | SCA OS0072 http://estseg.blogspot.com
On 06/09/2010 10:56, Piotr Jasiukajtis wrote:> Hi, > > I am looking for the ideas on how to check if the machine was under > high I/O pressure before it panicked (caused manually by an NMI). > By I/O I mean disks and ZFS stack. >Do you believe ZFS was a key component in the I/O pressure? I''ve CC''d zfs-discuss at opensolaris.org on my reply. If you think there was a lot of I/O happening, you could run: ::walk zio_root | ::zio -r This should give you an idea of the amount of ZIO going through ZFS. I would also be curious to look at the state of the pool(s) and the ZFS memory usage: ::spa -ev ::arc Kind regards, Jason
This is snv_128 x86.> ::archits = 39811943 misses = 630634 demand_data_hits = 29398113 demand_data_misses = 490754 demand_metadata_hits = 10413660 demand_metadata_misses = 133461 prefetch_data_hits = 0 prefetch_data_misses = 0 prefetch_metadata_hits = 170 prefetch_metadata_misses = 6419 mru_hits = 2933011 mru_ghost_hits = 43202 mfu_hits = 36878818 mfu_ghost_hits = 45361 deleted = 1299527 recycle_miss = 46526 mutex_miss = 355 evict_skip = 25539 evict_l2_cached = 0 evict_l2_eligible = 77011188736 evict_l2_ineligible = 76253184 hash_elements = 278135 hash_elements_max = 279843 hash_collisions = 1653518 hash_chains = 75135 hash_chain_max = 9 p = 4787 MB c = 5722 MB c_min = 715 MB c_max = 5722 MB size = 5428 MB hdr_size = 56535840 data_size = 5158287360 other_size = 477726560 l2_hits = 0 l2_misses = 0 l2_feeds = 0 l2_rw_clash = 0 l2_read_bytes = 0 l2_write_bytes = 0 l2_writes_sent = 0 l2_writes_done = 0 l2_writes_error = 0 l2_writes_hdr_miss = 0 l2_evict_lock_retry = 0 l2_evict_reading = 0 l2_free_on_write = 0 l2_abort_lowmem = 0 l2_cksum_bad = 0 l2_io_error = 0 l2_size = 0 l2_hdr_size = 0 memory_throttle_count = 0 arc_no_grow = 0 arc_tempreserve = 0 MB arc_meta_used = 1288 MB arc_meta_limit = 1430 MB arc_meta_max = 1288 MB> ::memstatPage Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 789865 3085 19% ZFS File Data 1406055 5492 34% Anon 396297 1548 9% Exec and libs 7178 28 0% Page cache 8428 32 0% Free (cachelist) 117928 460 3% Free (freelist) 1464224 5719 35% Total 4189975 16367 Physical 4189974 16367> ::spa -evADDR STATE NAME ffffff04f0eb4500 ACTIVE data ADDR STATE AUX DESCRIPTION ffffff04f2f52940 HEALTHY - root READ WRITE FREE CLAIM IOCTL OPS 0 0 0 0 0 BYTES 0 0 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff050a2fd980 HEALTHY - raidz READ WRITE FREE CLAIM IOCTL OPS 0x57090 0x37436a 0 0 0 BYTES 0x8207f3c00 0x22345d0800 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff050a2fa0c0 HEALTHY - /dev/dsk/c7t2d0s0 READ WRITE FREE CLAIM IOCTL OPS 0x4416e 0x105640 0 0 0x74326 BYTES 0x10909da00 0x45089d600 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff050a2fa700 HEALTHY - /dev/dsk/c7t3d0s0 READ WRITE FREE CLAIM IOCTL OPS 0x43fca 0x1055fa 0 0 0x74326 BYTES 0x108e14400 0x45087a400 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff050a2fad40 HEALTHY - /dev/dsk/c7t4d0s0 READ WRITE FREE CLAIM IOCTL OPS 0x44221 0x105533 0 0 0x74326 BYTES 0x108a56c00 0x4508c8a00 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff050a2fb380 HEALTHY - /dev/dsk/c7t5d0s0 READ WRITE FREE CLAIM IOCTL OPS 0x44195 0x105528 0 0 0x74325 BYTES 0x108b8c200 0x4508cfe00 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff050a2fb9c0 HEALTHY - /dev/dsk/c7t6d0s0 READ WRITE FREE CLAIM IOCTL OPS 0x441f3 0x10552c 0 0 0x74326 BYTES 0x108e84800 0x4508c7a00 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff050a2fc080 HEALTHY - /dev/dsk/c7t7d0s0 READ WRITE FREE CLAIM IOCTL OPS 0x43f34 0x105529 0 0 0x74326 BYTES 0x1080fc000 0x450869c00 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff050a2fc6c0 HEALTHY - /dev/dsk/c7t8d0s0 READ WRITE FREE CLAIM IOCTL OPS 0x43e8d 0x10559d 0 0 0x74326 BYTES 0x10833d000 0x4508a9200 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff050a2fcd00 HEALTHY - /dev/dsk/c7t9d0s0 READ WRITE FREE CLAIM IOCTL OPS 0x43aee 0x105671 0 0 0x74325 BYTES 0x10714f000 0x45089b600 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff050a2fd340 HEALTHY - /dev/dsk/c7t10d0s0 READ WRITE FREE CLAIM IOCTL OPS 0x442cf 0x105693 0 0 0x74325 BYTES 0x109338800 0x45086c200 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff04e6fcf080 ACTIVE rpool ffffff04e1c6dcc0 HEALTHY - root READ WRITE FREE CLAIM IOCTL OPS 0 0 0 0 0 BYTES 0 0 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff04e1c6d680 HEALTHY - mirror READ WRITE FREE CLAIM IOCTL OPS 0x3fc1d 0x169378 0 0 0 BYTES 0x2c0409e00 0x214e47c00 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff04e1c6d040 HEALTHY - /dev/dsk/c7t0d0s0 READ WRITE FREE CLAIM IOCTL OPS 0x1cc01 0xe8749 0 0 0x4915 BYTES 0x1cb5a6a00 0x215b96c00 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff04e92c2980 HEALTHY - /dev/dsk/c7t1d0s0 READ WRITE FREE CLAIM IOCTL OPS 0x1e3c5 0xe8556 0 0 0x4915 BYTES 0x1cfa84c00 0x215b96c00 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff04f0eb3a80 ACTIVE tmp_data ffffff050bb12d40 HEALTHY - root READ WRITE FREE CLAIM IOCTL OPS 0 0 0 0 0 BYTES 0 0 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0 ffffff050bb12700 HEALTHY - /dev/dsk/c7t11d0s0 READ WRITE FREE CLAIM IOCTL OPS 0x2dce 0x2d3c1 0 0 0x15a BYTES 0x2b37c400 0x4dcc81e00 0 0 0 EREAD 0 EWRITE 0 ECKSUM 0> ::walk zio_rootffffff05664b0328 ffffff04eb660060 ffffff04e96f9c88 ffffff05207bd658 ffffff05667fccb8 ffffff05678449a0 ffffff05678b6018 ffffff0568aef640 ffffff0566ece678 ffffff050afa09a0 ffffff055afef658> ::walk zio_root | ::zio -rADDRESS TYPE STAGE WAITER ffffff05664b0328 NULL CHECKSUM_VERIFY ffffff051bb13b00 ffffff05628fa680 WRITE VDEV_IO_START - ffffff0567d15370 WRITE VDEV_IO_START - ffffff0567409ce0 WRITE VDEV_IO_START - ffffff0566cbf968 WRITE VDEV_IO_START - ffffff056748cca8 WRITE VDEV_IO_START - ffffff055b184028 WRITE VDEV_IO_START - ffffff0567482328 WRITE VDEV_IO_START - ffffff0562f73658 WRITE VDEV_IO_START - ffffff04eb660060 NULL OPEN - ffffff04e96f9c88 NULL OPEN - ffffff05207bd658 NULL CHECKSUM_VERIFY ffffff001fe7fc60 ffffff055bc67060 WRITE VDEV_IO_START - ffffff0568160048 WRITE VDEV_IO_START - ffffff05661fbca8 WRITE VDEV_IO_START - ffffff0566edacc0 WRITE VDEV_IO_START - ffffff05665d5018 WRITE VDEV_IO_START - ffffff05667c3008 WRITE VDEV_IO_START - ffffff05664b39c0 WRITE VDEV_IO_START - ffffff051cea6010 WRITE VDEV_IO_START - ffffff051d333370 WRITE VDEV_IO_START - ffffff0521255048 WRITE VDEV_IO_START - This is not all output.> ::walk zio_root | ::zio -r ! wc -l7099 I am hitting this issue on 2 machines, both 128. The system is not response (ping still works) so I bet there is some kind of deadlock within ZFS. Were there any known ZFS related bugs similar to this one within b128? On Mon, Sep 6, 2010 at 12:13 PM, Jason Banham <jason.banham at oracle.com> wrote:> On 06/09/2010 10:56, Piotr Jasiukajtis wrote: >> >> Hi, >> >> I am looking for the ideas on how to check if the machine was under >> high I/O pressure before it panicked (caused manually by an NMI). >> By I/O I mean disks and ZFS stack. >> > > Do you believe ZFS was a key component in the I/O pressure? > I''ve CC''d zfs-discuss at opensolaris.org on my reply. > > If you think there was a lot of I/O happening, you could run: > > ::walk zio_root | ::zio -r > > This should give you an idea of the amount of ZIO going through ZFS. > I would also be curious to look at the state of the pool(s) and the > ZFS memory usage: > > ::spa -ev > ::arc > > > > > Kind regards, > > Jason > _______________________________________________ > mdb-discuss mailing list > mdb-discuss at opensolaris.org >-- Piotr Jasiukajtis | estibi | SCA OS0072 http://estseg.blogspot.com
Ok, now I know it''s not related to the I/O performance, but to the ZFS
itself.
At some time all 3 pools were locked in that way:
extended device statistics ---- errors ---
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b s/w h/w
trn tot device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 1
0 1 c8t0d0
0.0 0.0 0.0 0.0 0.0 8.0 0.0 0.0 0 100 0 0
0 0 c7t0d0
0.0 0.0 0.0 0.0 0.0 8.0 0.0 0.0 0 100 0 0
0 0 c7t1d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0
0 0 c7t2d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0
0 0 c7t3d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0
0 0 c7t4d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0
0 0 c7t5d0
0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0 100 0 0
0 0 c7t10d0
0.0 0.0 0.0 0.0 0.0 3.0 0.0 0.0 0 100 0 0
0 0 c7t11d0
^C
# zpool status
pool: data
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c7t2d0 ONLINE 0 0 0
c7t3d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c7t4d0 ONLINE 0 0 0
c7t5d0 ONLINE 0 0 0
errors: No known data errors
pool: rpool
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using ''zpool upgrade''. Once this is
done, the
pool will no longer be accessible on older software versions.
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c7t0d0s0 ONLINE 0 0 0
c7t1d0s0 ONLINE 0 0 0
errors: No known data errors
pool: tmp_data
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h1m, 0.74% done, 2h21m to go
config:
NAME STATE READ WRITE CKSUM
tmp_data ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c7t11d0 ONLINE 0 0 0
c7t10d0 ONLINE 0 0 0 2.07G resilvered
errors: No known data errors
Resilvering tmp_data is not related. I did zpool attach manually.
On Tue, Sep 7, 2010 at 12:39 PM, Piotr Jasiukajtis <estseg at gmail.com>
wrote:> This is snv_128 x86.
>
>> ::arc
> hits ? ? ? ? ? ? ? ? ? ? ?= ?39811943
> misses ? ? ? ? ? ? ? ? ? ?= ? ?630634
> demand_data_hits ? ? ? ? ?= ?29398113
> demand_data_misses ? ? ? ?= ? ?490754
> demand_metadata_hits ? ? ?= ?10413660
> demand_metadata_misses ? ?= ? ?133461
> prefetch_data_hits ? ? ? ?= ? ? ? ? 0
> prefetch_data_misses ? ? ?= ? ? ? ? 0
> prefetch_metadata_hits ? ?= ? ? ? 170
> prefetch_metadata_misses ?= ? ? ?6419
> mru_hits ? ? ? ? ? ? ? ? ?= ? 2933011
> mru_ghost_hits ? ? ? ? ? ?= ? ? 43202
> mfu_hits ? ? ? ? ? ? ? ? ?= ?36878818
> mfu_ghost_hits ? ? ? ? ? ?= ? ? 45361
> deleted ? ? ? ? ? ? ? ? ? = ? 1299527
> recycle_miss ? ? ? ? ? ? ?= ? ? 46526
> mutex_miss ? ? ? ? ? ? ? ?= ? ? ? 355
> evict_skip ? ? ? ? ? ? ? ?= ? ? 25539
> evict_l2_cached ? ? ? ? ? = ? ? ? ? 0
> evict_l2_eligible ? ? ? ? = 77011188736
> evict_l2_ineligible ? ? ? = ?76253184
> hash_elements ? ? ? ? ? ? = ? ?278135
> hash_elements_max ? ? ? ? = ? ?279843
> hash_collisions ? ? ? ? ? = ? 1653518
> hash_chains ? ? ? ? ? ? ? = ? ? 75135
> hash_chain_max ? ? ? ? ? ?= ? ? ? ? 9
> p ? ? ? ? ? ? ? ? ? ? ? ? = ? ? ?4787 MB
> c ? ? ? ? ? ? ? ? ? ? ? ? = ? ? ?5722 MB
> c_min ? ? ? ? ? ? ? ? ? ? = ? ? ? 715 MB
> c_max ? ? ? ? ? ? ? ? ? ? = ? ? ?5722 MB
> size ? ? ? ? ? ? ? ? ? ? ?= ? ? ?5428 MB
> hdr_size ? ? ? ? ? ? ? ? ?= ?56535840
> data_size ? ? ? ? ? ? ? ? = 5158287360
> other_size ? ? ? ? ? ? ? ?= 477726560
> l2_hits ? ? ? ? ? ? ? ? ? = ? ? ? ? 0
> l2_misses ? ? ? ? ? ? ? ? = ? ? ? ? 0
> l2_feeds ? ? ? ? ? ? ? ? ?= ? ? ? ? 0
> l2_rw_clash ? ? ? ? ? ? ? = ? ? ? ? 0
> l2_read_bytes ? ? ? ? ? ? = ? ? ? ? 0
> l2_write_bytes ? ? ? ? ? ?= ? ? ? ? 0
> l2_writes_sent ? ? ? ? ? ?= ? ? ? ? 0
> l2_writes_done ? ? ? ? ? ?= ? ? ? ? 0
> l2_writes_error ? ? ? ? ? = ? ? ? ? 0
> l2_writes_hdr_miss ? ? ? ?= ? ? ? ? 0
> l2_evict_lock_retry ? ? ? = ? ? ? ? 0
> l2_evict_reading ? ? ? ? ?= ? ? ? ? 0
> l2_free_on_write ? ? ? ? ?= ? ? ? ? 0
> l2_abort_lowmem ? ? ? ? ? = ? ? ? ? 0
> l2_cksum_bad ? ? ? ? ? ? ?= ? ? ? ? 0
> l2_io_error ? ? ? ? ? ? ? = ? ? ? ? 0
> l2_size ? ? ? ? ? ? ? ? ? = ? ? ? ? 0
> l2_hdr_size ? ? ? ? ? ? ? = ? ? ? ? 0
> memory_throttle_count ? ? = ? ? ? ? 0
> arc_no_grow ? ? ? ? ? ? ? = ? ? ? ? 0
> arc_tempreserve ? ? ? ? ? = ? ? ? ? 0 MB
> arc_meta_used ? ? ? ? ? ? = ? ? ?1288 MB
> arc_meta_limit ? ? ? ? ? ?= ? ? ?1430 MB
> arc_meta_max ? ? ? ? ? ? ?= ? ? ?1288 MB
>
>> ::memstat
> Page Summary ? ? ? ? ? ? ? ?Pages ? ? ? ? ? ? ? ?MB ?%Tot
> ------------ ? ? ---------------- ?---------------- ?----
> Kernel ? ? ? ? ? ? ? ? ? ? 789865 ? ? ? ? ? ? ?3085 ? 19%
> ZFS File Data ? ? ? ? ? ? 1406055 ? ? ? ? ? ? ?5492 ? 34%
> Anon ? ? ? ? ? ? ? ? ? ? ? 396297 ? ? ? ? ? ? ?1548 ? ?9%
> Exec and libs ? ? ? ? ? ? ? ?7178 ? ? ? ? ? ? ? ?28 ? ?0%
> Page cache ? ? ? ? ? ? ? ? ? 8428 ? ? ? ? ? ? ? ?32 ? ?0%
> Free (cachelist) ? ? ? ? ? 117928 ? ? ? ? ? ? ? 460 ? ?3%
> Free (freelist) ? ? ? ? ? 1464224 ? ? ? ? ? ? ?5719 ? 35%
>
> Total ? ? ? ? ? ? ? ? ? ? 4189975 ? ? ? ? ? ? 16367
> Physical ? ? ? ? ? ? ? ? ?4189974 ? ? ? ? ? ? 16367
>
>
>> ::spa -ev
> ADDR ? ? ? ? ? ? ? ? STATE NAME
> ffffff04f0eb4500 ? ?ACTIVE data
>
> ? ?ADDR ? ? ? ? ? ? STATE ? ? AUX ? ? ? ? ?DESCRIPTION
> ? ?ffffff04f2f52940 HEALTHY ? - ? ? ? ? ? ?root
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? ? ? ? 0 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?BYTES ? ? ? ? ? ? 0 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff050a2fd980 HEALTHY ? - ? ? ? ? ? ? ?raidz
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x57090 ? ? 0x37436a ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?BYTES ? 0x8207f3c00 ?0x22345d0800 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ?
? ?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff050a2fa0c0 HEALTHY ? - ? ? ? ? ? ? ? ?/dev/dsk/c7t2d0s0
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x4416e ? ? 0x105640 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ?
?0x74326
> ? ? ? ?BYTES ? 0x10909da00 ?0x45089d600 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff050a2fa700 HEALTHY ? - ? ? ? ? ? ? ? ?/dev/dsk/c7t3d0s0
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x43fca ? ? 0x1055fa ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ?
?0x74326
> ? ? ? ?BYTES ? 0x108e14400 ?0x45087a400 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff050a2fad40 HEALTHY ? - ? ? ? ? ? ? ? ?/dev/dsk/c7t4d0s0
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x44221 ? ? 0x105533 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ?
?0x74326
> ? ? ? ?BYTES ? 0x108a56c00 ?0x4508c8a00 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff050a2fb380 HEALTHY ? - ? ? ? ? ? ? ? ?/dev/dsk/c7t5d0s0
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x44195 ? ? 0x105528 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ?
?0x74325
> ? ? ? ?BYTES ? 0x108b8c200 ?0x4508cfe00 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff050a2fb9c0 HEALTHY ? - ? ? ? ? ? ? ? ?/dev/dsk/c7t6d0s0
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x441f3 ? ? 0x10552c ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ?
?0x74326
> ? ? ? ?BYTES ? 0x108e84800 ?0x4508c7a00 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff050a2fc080 HEALTHY ? - ? ? ? ? ? ? ? ?/dev/dsk/c7t7d0s0
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x43f34 ? ? 0x105529 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ?
?0x74326
> ? ? ? ?BYTES ? 0x1080fc000 ?0x450869c00 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff050a2fc6c0 HEALTHY ? - ? ? ? ? ? ? ? ?/dev/dsk/c7t8d0s0
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x43e8d ? ? 0x10559d ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ?
?0x74326
> ? ? ? ?BYTES ? 0x10833d000 ?0x4508a9200 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff050a2fcd00 HEALTHY ? - ? ? ? ? ? ? ? ?/dev/dsk/c7t9d0s0
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x43aee ? ? 0x105671 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ?
?0x74325
> ? ? ? ?BYTES ? 0x10714f000 ?0x45089b600 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff050a2fd340 HEALTHY ? - ? ? ? ? ? ? ? ?/dev/dsk/c7t10d0s0
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x442cf ? ? 0x105693 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ?
?0x74325
> ? ? ? ?BYTES ? 0x109338800 ?0x45086c200 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ffffff04e6fcf080 ? ?ACTIVE rpool
>
> ? ?ffffff04e1c6dcc0 HEALTHY ? - ? ? ? ? ? ?root
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? ? ? ? 0 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?BYTES ? ? ? ? ? ? 0 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff04e1c6d680 HEALTHY ? - ? ? ? ? ? ? ?mirror
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x3fc1d ? ? 0x169378 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?BYTES ? 0x2c0409e00 ?0x214e47c00 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff04e1c6d040 HEALTHY ? - ? ? ? ? ? ? ? ?/dev/dsk/c7t0d0s0
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x1cc01 ? ? ?0xe8749 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ?
0x4915
> ? ? ? ?BYTES ? 0x1cb5a6a00 ?0x215b96c00 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff04e92c2980 HEALTHY ? - ? ? ? ? ? ? ? ?/dev/dsk/c7t1d0s0
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? 0x1e3c5 ? ? ?0xe8556 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ?
0x4915
> ? ? ? ?BYTES ? 0x1cfa84c00 ?0x215b96c00 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ffffff04f0eb3a80 ? ?ACTIVE tmp_data
>
> ? ?ffffff050bb12d40 HEALTHY ? - ? ? ? ? ? ?root
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? ? ? ? 0 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?BYTES ? ? ? ? ? ? 0 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
> ? ?ffffff050bb12700 HEALTHY ? - ? ? ? ? ? ? ?/dev/dsk/c7t11d0s0
>
> ? ? ? ? ? ? ? ? ? ? ? READ ? ? ? ?WRITE ? ? ? ? FREE ? ? ? ?CLAIM ? ? ?
?IOCTL
> ? ? ? ?OPS ? ? ? ? ?0x2dce ? ? ?0x2d3c1 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ?
?0x15a
> ? ? ? ?BYTES ? ?0x2b37c400 ?0x4dcc81e00 ? ? ? ? ? ?0 ? ? ? ? ? ?0 ? ? ? ? ?
?0
> ? ? ? ?EREAD ? ? ? ? ? ? 0
> ? ? ? ?EWRITE ? ? ? ? ? ?0
> ? ? ? ?ECKSUM ? ? ? ? ? ?0
>
>
>> ::walk zio_root
> ffffff05664b0328
> ffffff04eb660060
> ffffff04e96f9c88
> ffffff05207bd658
> ffffff05667fccb8
> ffffff05678449a0
> ffffff05678b6018
> ffffff0568aef640
> ffffff0566ece678
> ffffff050afa09a0
> ffffff055afef658
>
>> ::walk zio_root | ::zio -r
> ADDRESS ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?TYPE ?STAGE ? ? ? ? ? ?WAITER
> ffffff05664b0328 ? ? ? ? ? ? ? ? ? ? ? ? NULL ?CHECKSUM_VERIFY
?ffffff051bb13b00
> ?ffffff05628fa680 ? ? ? ? ? ? ? ? ? ? ? ?WRITE VDEV_IO_START ? ?-
> ?ffffff0567d15370 ? ? ? ? ? ? ? ? ? ? ? WRITE VDEV_IO_START ? ?-
> ? ffffff0567409ce0 ? ? ? ? ? ? ? ? ? ? ?WRITE VDEV_IO_START ? ?-
> ? ffffff0566cbf968 ? ? ? ? ? ? ? ? ? ? ?WRITE VDEV_IO_START ? ?-
> ?ffffff056748cca8 ? ? ? ? ? ? ? ? ? ? ? ?WRITE VDEV_IO_START ? ?-
> ?ffffff055b184028 ? ? ? ? ? ? ? ? ? ? ? WRITE VDEV_IO_START ? ?-
> ? ffffff0567482328 ? ? ? ? ? ? ? ? ? ? ?WRITE VDEV_IO_START ? ?-
> ? ffffff0562f73658 ? ? ? ? ? ? ? ? ? ? ?WRITE VDEV_IO_START ? ?-
> ffffff04eb660060 ? ? ? ? ? ? ? ? ? ? ? ? NULL ?OPEN ? ? ? ? ? ? -
> ffffff04e96f9c88 ? ? ? ? ? ? ? ? ? ? ? ? NULL ?OPEN ? ? ? ? ? ? -
> ffffff05207bd658 ? ? ? ? ? ? ? ? ? ? ? ? NULL ?CHECKSUM_VERIFY
?ffffff001fe7fc60
> ?ffffff055bc67060 ? ? ? ? ? ? ? ? ? ? ? ?WRITE VDEV_IO_START ? ?-
> ?ffffff0568160048 ? ? ? ? ? ? ? ? ? ? ? WRITE VDEV_IO_START ? ?-
> ? ffffff05661fbca8 ? ? ? ? ? ? ? ? ? ? ?WRITE VDEV_IO_START ? ?-
> ? ffffff0566edacc0 ? ? ? ? ? ? ? ? ? ? ?WRITE VDEV_IO_START ? ?-
> ?ffffff05665d5018 ? ? ? ? ? ? ? ? ? ? ? WRITE VDEV_IO_START ? ?-
> ? ffffff05667c3008 ? ? ? ? ? ? ? ? ? ? ?WRITE VDEV_IO_START ? ?-
> ? ffffff05664b39c0 ? ? ? ? ? ? ? ? ? ? ?WRITE VDEV_IO_START ? ?-
> ?ffffff051cea6010 ? ? ? ? ? ? ? ? ? ? ? WRITE VDEV_IO_START ? ?-
> ? ffffff051d333370 ? ? ? ? ? ? ? ? ? ? ?WRITE VDEV_IO_START ? ?-
> ? ?ffffff0521255048 ? ? ? ? ? ? ? ? ? ? WRITE VDEV_IO_START ? ?-
>
> This is not all output.
>
>> ::walk zio_root | ::zio -r ! wc -l
> 7099
>
> I am hitting this issue on 2 machines, both 128.
> The system is not response (ping still works) so I bet there is some
> kind of deadlock within ZFS.
>
> Were there any known ZFS related bugs similar to this one within b128?
>
> On Mon, Sep 6, 2010 at 12:13 PM, Jason Banham <jason.banham at
oracle.com> wrote:
>> On 06/09/2010 10:56, Piotr Jasiukajtis wrote:
>>>
>>> Hi,
>>>
>>> I am looking for the ideas on how to check if the machine was under
>>> high I/O pressure before it panicked (caused manually by an NMI).
>>> By I/O I mean disks and ZFS stack.
>>>
>>
>> Do you believe ZFS was a key component in the I/O pressure?
>> I''ve CC''d zfs-discuss at opensolaris.org on my reply.
>>
>> If you think there was a lot of I/O happening, you could run:
>>
>> ::walk zio_root | ::zio -r
>>
>> This should give you an idea of the amount of ZIO going through ZFS.
>> I would also be curious to look at the state of the pool(s) and the
>> ZFS memory usage:
>>
>> ::spa -ev
>> ::arc
>>
>>
>>
>>
>> Kind regards,
>>
>> Jason
>> _______________________________________________
>> mdb-discuss mailing list
>> mdb-discuss at opensolaris.org
>>
>
>
>
> --
> Piotr Jasiukajtis | estibi | SCA OS0072
> http://estseg.blogspot.com
>
--
Piotr Jasiukajtis | estibi | SCA OS0072
http://estseg.blogspot.com
On 9/10/10 4:16 PM, Piotr Jasiukajtis wrote:> Ok, now I know it''s not related to the I/O performance, but to the ZFS itself. > > At some time all 3 pools were locked in that way: > > extended device statistics ---- errors --- > r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b s/w h/w > trn tot device > 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 1 > 0 1 c8t0d0 > 0.0 0.0 0.0 0.0 0.0 8.0 0.0 0.0 0 100 0 0 > 0 0 c7t0d0Nope, most likely your disks or disk controller/driver. Note that you have 8 outstanding I/O requests that aren''t being serviced. Look in your syslog, and I bet you''ll see I/O timeout errors. I have seen this before with Western Digital disks attached to an LSI controller using the mpt driver. There was a lot of work diagnosing it, see the list archives - an /etc/system change fixed it for me (set xpv_psm:xen_support_msi = -1), but I was using a xen kernel. Note that replacing my disks with larger Seagate ones made the problem go away as well.
I don''t have any errors from fmdump or syslog. The machine is SUN FIRE X4275 I don''t use mpt or lsi drivers. It could be a bug in a driver since I see this on 2 the same machines. On Fri, Sep 10, 2010 at 9:51 PM, Carson Gaspar <carson at taltos.org> wrote:> On 9/10/10 4:16 PM, Piotr Jasiukajtis wrote: >> >> Ok, now I know it''s not related to the I/O performance, but to the ZFS >> itself. >> >> At some time all 3 pools were locked in that way: >> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? extended device statistics ? ? ? ---- errors >> --- >> ? ? r/s ? ?w/s ? kr/s ? kw/s wait actv wsvc_t asvc_t ?%w ?%b s/w h/w >> trn tot device >> ? ? 0.0 ? ?0.0 ? ?0.0 ? ?0.0 ?0.0 ?0.0 ? ?0.0 ? ?0.0 ? 0 ? 0 ? 0 ? 1 >> 0 ? 1 c8t0d0 >> ? ? 0.0 ? ?0.0 ? ?0.0 ? ?0.0 ?0.0 ?8.0 ? ?0.0 ? ?0.0 ? 0 100 ? 0 ? 0 >> 0 ? 0 c7t0d0 > > Nope, most likely your disks or disk controller/driver. Note that you have 8 > outstanding I/O requests that aren''t being serviced. Look in your syslog, > and I bet you''ll see I/O timeout errors. I have seen this before with > Western Digital disks attached to an LSI controller using the mpt driver. > There was a lot of work diagnosing it, see the list archives - an > /etc/system change fixed it for me (set xpv_psm:xen_support_msi = -1), but I > was using a xen kernel. Note that replacing my disks with larger Seagate > ones made the problem go away as well. > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- Piotr Jasiukajtis | estibi | SCA OS0072 http://estseg.blogspot.com
You are both right. More below... On Sep 10, 2010, at 2:06 PM, Piotr Jasiukajtis wrote:> I don''t have any errors from fmdump or syslog. > The machine is SUN FIRE X4275 I don''t use mpt or lsi drivers. > It could be a bug in a driver since I see this on 2 the same machines. > > On Fri, Sep 10, 2010 at 9:51 PM, Carson Gaspar <carson at taltos.org> wrote: >> On 9/10/10 4:16 PM, Piotr Jasiukajtis wrote: >>> >>> Ok, now I know it''s not related to the I/O performance, but to the ZFS >>> itself. >>> >>> At some time all 3 pools were locked in that way: >>> >>> extended device statistics ---- errors >>> --- >>> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn tot device >>> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 1 0 1 c8t0d0 >>> 0.0 0.0 0.0 0.0 0.0 8.0 0.0 0.0 0 100 0 0 0 0 c7t0d0 >> >> Nope, most likely your disks or disk controller/driver. Note that you have 8 >> outstanding I/O requests that aren''t being serviced. Look in your syslog, >> and I bet you''ll see I/O timeout errors. I have seen this before with >> Western Digital disks attached to an LSI controller using the mpt driver. >> There was a lot of work diagnosing it, see the list archives - an >> /etc/system change fixed it for me (set xpv_psm:xen_support_msi = -1), but I >> was using a xen kernel. Note that replacing my disks with larger Seagate >> ones made the problem go away as well.In this case, the diagnosis that I/Os are stuck at the drive, not being serviced is correct. This is clearly visible as actv>0, asvc_t==0, and the derived %b == 100% However, the error reports are also 0 for the affected devices: s/w, h/w, and trn. In many cases where we see I/O timeouts and devices aborting commands, we will see these logged as transport (trn) errors. For iostat, these errors are reported as since-boot, not per-sample period, so we know that whatever is getting stuck isn''t getting unstuck. The symptom we see with questionable devices in the HBA-to-disk path is hundreds, thousands, or millions of transport errors reported. Next question: what does the software stack look like? I knew the sd driver intimately at one time (pictures were in the Enquirer :-) and it will retry and send resets that will ultimately get logged. In this case, we know that at least one hard error was returned for c8t0d0, so there is a ereport somewhere with the details, try "fmdump -eV" This is not a ZFS bug and cannot be fixed at the ZFS layer. -- richard -- OpenStorage Summit, October 25-27, Palo Alto, CA http://nexenta-summit2010.eventbrite.com Richard Elling richard at nexenta.com +1-760-896-4422 Enterprise class storage for everyone www.nexenta.com