search for: zio_don

I have been using iosoop script (see http://www.opensolaris.org/os/community/dtrace/scripts/) written by Brendan Gregg to look at the IO operations of my application. When I was running my test-program on a UFS filesystem I could see both read and write operations like: UID PID D BLOCK SIZE COMM PATHNAME 203803 4436 R 6016592 16384 diskio <none> 203803 4436 W 3448432

...P DVA[0]=<0:770b24 000:4000> DVA[1]=<0:dfa984000:4000> fletcher4 uncompressed LE contiguous birth=260276 fill=1 cksum=1:1000:800800:2ab2ab000): error 5 Apr 23 02:02:22 SERVER144 unix: [ID 100000 kern.notice] Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1a40 zfs:zio_done+17c () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1a60 zfs:zio_next_stage+b3 () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1ab0 zfs:zio_wait_for_children+5d () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1ad0 zfs:zi...

...Free (cachelist) 11554 45 9% Free (freelist) 4742 18 4% Total 128910 503 Physical 128909 503 > $C db95bae4 vpanic(f9f95858, d7a91388, d7a913b8, f9f94660, 0, 0) db95bc70 zio_done+0x122(dbbf8880) db95bc8c zio_next_stage+0x66(dbbf8880) db95bcac zio_wait_for_children+0x46(dbbf8880, 11, dbbf8a70) db95bcc0 zio_wait_children_done+0x18(dbbf8880) db95bcd8 zio_next_stage+0x66(dbbf8880) db95bd10 zio_vdev_io_assess+0x11a(dbbf8880) db95bd24 zio_next_stage+0x66(dbbf8880) db95bd64 vdev_...

Now at 36 hours since zdb process start and: PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 827 root 4936M 4931M sleep 59 0 0:50:47 0.2% zdb/209 Idling at 0.2% processor for nearly the past 24 hours... feels very stuck. Thoughts on how to determine where and why? -- This message posted from opensolaris.org

...otected ZFS: I/O failure (write on <unknown> off 0: zio 6001f3f2340 [L0 DMU dnode] 4000L/a00P DVA[0]=<0:54e000:a00> DVA[1]=<0:d000ab800:a00> fletcher4 lzjb BE contiguous birth=597 fill=31 cksum=97d1c4f896:dd5b900a7ec8:b89066e037beda:7021cc17c66f4713): error 6 000002a10103b880 zfs:zio_done+29c (0, 10, 6000e361400, 600202bfb80, 6001f3f2340, 0) %l0-3: 0000060003233b40 00000600109c1740 000000007085a218 0000000000000006 %l4-7: 0000000000000002 0000000000000255 0000000000000006 00000000018bf400 000002a10103b940 zfs:zio_vdev_io_assess+178 (6001f3f2340, 8000, 10, 0, 0, 10) %l0-3: 000...

I executed sync just before this happened.... ultra:ultra# mdb -k unix.0 vmcore.0 Loading modules: [ unix krtld genunix specfs dtrace ufs sd pcipsy md ip sctp usba fctl nca crypto zfs random nfs ptm cpc fcip sppp lofs ] > $c vpanic(7b653bd8, 7036fca0, 7036fc70, 7b652990, 0, 60002d0b480) zio_done+0x284(60002d0b480, 0, a8, 7036fca0, 0, 60000b08d80) zio_vdev_io_assess+0x178(60002d0b480, 8000, 10, 0, 0, 10) taskq_thread+0x1a4(6000080e210, 6000080e1b8, 50001, 2e6a0862848b, 2a10071daca, 2a10071dac8) thread_start+4(6000080e1b8, 0, 0, 0, 0, 0) > there may of been a single dead drive in 2 r...

...: bad checksum (read on off 0: zio 60015f26a40 [L0 packed nvlist] 4000L/400P DVA[0]=&lt;0:4000b7000:400&gt; DVA[1]=&lt;0:4600063800:400&gt; fletcher4 lzjb BE contiguous birth=3865837 fill=1 cksum=6e858ee992:2cad0296498c:cae1d40ad3d38:2e1daaef734ef08): error 50 000002a100b69880 zfs:zio_done+29c (0, 10, 60015f26a98, 60012e062c0, 60015f26a40, 0) %l0-3: 0000060015d04040 00000600160ffac0 00000000707de1c0 0000000000000032 %l4-7: 0000000000000001 0000000000000002 0000000000000032 0000060015c05118 000002a100b69940 zfs:zio_vdev_io_assess+178 (60015f26a40, 8000, 10, 0, 0, 10) %l0-3: 00000...

...out>) at lib/libzpool/build-kernel/arc.c:735 #3 0x0000000000457f32 in arc_access (buf=0x2aaabe2930c0, hash_lock=0x592c30) at lib/libzpool/build-kernel/arc.c:1637 #4 0x0000000000458ff9 in arc_read_done (zio=0x2aaabcfa4ed0) at lib/libzpool/build-kernel/arc.c:1850 #5 0x000000000044fb9f in zio_done (zio=0x2aaabcfa4ed0) at lib/libzpool/build-kernel/zio.c:868 #6 0x00000000004527f0 in zio_vdev_io_assess (zio=0x2aaabcfa4ed0) at lib/libzpool/build-kernel/zio.c:1491 #7 0x0000000000466ecf in taskq_thread (arg=<value optimized out>) at lib/libsolkerncompat/taskq.c:160 #8 0x00002afcd74273...

...d=2a1011d3cc0: ZFS: I/O failure (write on <unknown> off 0: zio 60007432bc0 [L0 unallocated] 4000L/400P DVA[0]=<0:b000:400> DVA[1]=<0:120a000:400> fletcher4 lzjb BE contiguous birth=6 fill=0 cksum=672165b9e7:328e78ae25fd:ed007c9008f5f:34c05b10900b668): error 6 000002a1011d3740 zfs:zio_done+284 (60007432bc0, 0, a8, 7035dbf0, 0, 60006fa9700) %l0-3: 00000600036d4b40 000000007035d800 0000000000000006 0000000000000006 %l4-7: 000000007bb9a278 0000000000000002 0000000000000006 0000000000000006 000002a1011d3940 zfs:zio_vdev_io_assess+178 (60007432bc0, 8000, 10, 0, 0, 10) %l0-3: 000000...

...[L0 unallocated] vdev=0 offset=8500e8200 size=200L/200P/200A fletcher4 uncompressed LE contiguous birth=41837 fill=0 cksum=bcd3026a:5d6fdb3032:174cfede75e7:3e7752513e700): error 6 Nov 19 09:09:34 fugly unix: [ID 100000 kern.notice] Nov 19 09:09:34 fugly genunix: [ID 353471 kern.notice] d51e5c7c zfs:zio_done+199 (d53a5600) Nov 19 09:09:34 fugly genunix: [ID 353471 kern.notice] d51e5c9c zfs:zio_next_stage+73 (d53a5600) Nov 19 09:09:34 fugly genunix: [ID 353471 kern.notice] d51e5cbc zfs:zio_wait_for_children+58 (d53a5600, 13, d53a5) Nov 19 09:09:34 fugly genunix: [ID 353471 kern.notice] d51e5cdc zfs:zio...

...L0 ZIL intent log] 2000L/2000P DVA[0]=<0:25786c7000:2800> zilog uncompressed LE contiguous birth=1468445 fill=0 cksum=4392a2279563047e:1b7716cbbf370c72:ac: 6b): error 5 May 9 16:47:43 sol unix: [ID 100000 kern.notice] May 9 16:47:43 sol genunix: [ID 655072 kern.notice] fffffe8000581a00 zfs:zio_done+2fc () May 9 16:47:43 sol genunix: [ID 655072 kern.notice] fffffe8000581a30 zfs:zio_next_stage+11e () May 9 16:47:43 sol genunix: [ID 655072 kern.notice] fffffe8000581a80 zfs:zio_wait_for_children+5e () May 9 16:47:43 sol genunix: [ID 655072 kern.notice] fffffe8000581aa0 zfs:zio_wait_children_d...