search for: taskq_thread

Now at 36 hours since zdb process start and: PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 827 root 4936M 4931M sleep 59 0 0:50:47 0.2% zdb/209 Idling at 0.2% processor for nearly the past 24 hours... feels very stuck. Thoughts on how to determine where and why? -- This message posted from opensolaris.org

I''ve recently re-installed an X4500 running Nevada b109 and have been experiencing ZFS lock ups regularly (perhaps once every 2-3 days). The machine is a backup server and receives hourly ZFS snapshots from another thumper - as such, the amount of zfs activity tends to be reasonably high. After about 48 - 72 hours, the file system seems to lock up and I''m unable to do anything

...b3 () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1bb0 zfs:vdev_mirror_io_done+2af () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1bd0 zfs:zio_vdev_io_done+26 () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1c60 genunix:taskq_thread+1a7 () Apr 23 02:02:23 SERVER144 genunix: [ID 655072 kern.notice] ffffff0017fa1c70 unix:thread_start+8 () Apr 23 02:02:23 SERVER144 unix: [ID 100000 kern.notice] Apr 23 02:02:23 SERVER144 genunix: [ID 672855 kern.notice] syncing file systems... Apr 23 02:02:23 SERVER144 genunix: [ID 433738 kern.not...

...t_for_children+0x46(dbbf8880, 11, dbbf8a70) db95bcc0 zio_wait_children_done+0x18(dbbf8880) db95bcd8 zio_next_stage+0x66(dbbf8880) db95bd10 zio_vdev_io_assess+0x11a(dbbf8880) db95bd24 zio_next_stage+0x66(dbbf8880) db95bd64 vdev_mirror_io_done+0x289(dbbf8880) db95bd78 zio_vdev_io_done+0x25() db95bdc8 taskq_thread+0x176(d430a290, 0) db95bdd8 thread_start+8() bash-3.00# zpool status -v pool: backup state: FAULTED status: One or more devices could not be used because the the label is missing or invalid. There are insufficient replicas for the pool to continue functioning. action: Destroy a...

...00000255 0000000000000006 00000000018bf400 000002a10103b940 zfs:zio_vdev_io_assess+178 (6001f3f2340, 8000, 10, 0, 0, 10) %l0-3: 0000000000000002 0000000000000002 0000000000000000 0000000000000006 %l4-7: 0000000000000010 0000000000000001 0000000000000000 0000000000000000 000002a10103ba00 genunix:taskq_thread+1a4 (60003929228, 600039291d0, 50001, 507428728c, 2a10103baca, 2a10103bac8) %l0-3: 0000000000010000 00000600039291f8 0000060003929200 0000060003929202 %l4-7: 0000060003927a50 0000000000000002 0000000000000000 00000600039291f0 syncing file systems... done Is there some magic zfs-option such...

...AP: type=e (#pf Page fault) rp=ffffff0010a5e920 addr=30 > occurred in module "zfs" due to a NULL pointer dereference > panicstack = unix:die+dd () | unix:trap+1799 () | unix:cmntrap+e6 () | > zfs:ddt_phys_decref+c () | zfs:zio_ddt_free+5c () | zfs:zio_execute+8d > () | genunix:taskq_thread+285 () | unix:thread_start+8 () | > crashtime = 1328314064 > panic-time = February 4, 2012 04:07:44 AM MSK MSK > (end fault-list[0]) > > fault-status = 0x1 > severity = Major > __ttl = 0x1 > __tod = 0x4f2c77a8 0x37734060 > > And here''s some dmesg leading up to...

I just had a quick play with gzip compression on a filesystem and the result was the machine grinding to a halt while copying some large (.wav) files to it from another filesystem in the same pool. The system became very unresponsive, taking several seconds to echo keystrokes. The box is a maxed out AMD QuadFX, so it should have plenty of grunt for this. Comments? Ian

...s:vdev_disk_io_done+29 () Oct 10 17:24:04 zetta genunix: [ID 655072 kern.notice] ffffff0007f1bbb0 zfs:vdev_io_done+1d () Oct 10 17:24:04 zetta genunix: [ID 655072 kern.notice] ffffff0007f1bbd0 zfs:zio_vdev_io_done+1b () Oct 10 17:24:04 zetta genunix: [ID 655072 kern.notice] ffffff0007f1bc60 genunix:taskq_thread+1a7 () Oct 10 17:24:04 zetta genunix: [ID 655072 kern.notice] ffffff0007f1bc70 unix:thread_start+8 () Oct 10 17:24:04 zetta unix: [ID 100000 kern.notice] Oct 10 17:24:04 zetta genunix: [ID 672855 kern.notice] syncing file systems... Oct 10 17:24:04 zetta genunix: [ID 733762 kern.notice] 26 Oct 10...

...d genunix specfs dtrace ufs sd pcipsy md ip sctp usba fctl nca crypto zfs random nfs ptm cpc fcip sppp lofs ] > $c vpanic(7b653bd8, 7036fca0, 7036fc70, 7b652990, 0, 60002d0b480) zio_done+0x284(60002d0b480, 0, a8, 7036fca0, 0, 60000b08d80) zio_vdev_io_assess+0x178(60002d0b480, 8000, 10, 0, 0, 10) taskq_thread+0x1a4(6000080e210, 6000080e1b8, 50001, 2e6a0862848b, 2a10071daca, 2a10071dac8) thread_start+4(6000080e1b8, 0, 0, 0, 0, 0) > there may of been a single dead drive in 2 raidz vdevs. ultra:ultra# zpool status pool: data state: DEGRADED status: One or more devices could not be opened. Suffi...

...3a6 () ffffff00044698f0 zfs:metaslab_activate+93 () ffffff00044699b0 zfs:metaslab_group_alloc+24e () ffffff0004469a70 zfs:metaslab_alloc_dva+200 () ffffff0004469b30 zfs:metaslab_alloc+156 () ffffff0004469b90 zfs:zio_dva_allocate+10d () ffffff0004469bd0 zfs:zio_execute+bb () ffffff0004469c60 genunix:taskq_thread+1cb () ffffff0004469c70 unix:thread_start+8 () panic: entering debugger (continue to save dump) The stress test zfs_force_umount_stress panics, see 2232. After rebooting, there is a pool created by zfs_force_umount_stress, then running ''zfs destroy'' panics. # df -h Filesystem...

...c48610c0, ffffff00c679e250, ffffff00c6c63be0, ffffff0002377bc4) xpvd`xpvd_post_event+0x24(ffffff00c660d680, ffffff00c679e250, ffffff00c6c63be0, ffffff0002377bc4) ndi_post_event+0x2d(ffffff00c679e250, ffffff00c679e250, ffffff00c6c63be0, ffffff0002377bc4) i_xvdi_oestate_handler+0x74(ffffff00c679e250) taskq_thread+0x1a7(ffffff00c6cda0e8) thread_start+8() xdf_oe_change() is called with an "XenbusStateClosed" event! That is: Using config file "./solaris-install2". Started domain solaris-install2 Loading kmdb... Welcome to kmdb Loaded modules: [ unix krtl...

...lback+0x193 unix`hat_unload+0x41 unix`segkmem_free_vn+0x6f unix`segkmem_zio_free+0x27 genunix`vmem_xfree+0x104 genunix`vmem_free+0x29 genunix`kmem_slab_destroy+0x87 genunix`kmem_slab_free+0x2bb genunix`kmem_magazine_destroy+0x39a genunix`kmem_depot_ws_reap+0x66 genunix`taskq_thread+0x285 unix`thread_start+0x8 3221701 This happens in the sched (pid 0) process. My fsstat one looks like this: # fsstat /content 1 new name name attr attr lookup rddir read read write write file remov chng get set ops ops ops bytes ops bytes 0 0 0 664 0...

...00000002 0000000000000032 0000060015c05118 000002a100b69940 zfs:zio_vdev_io_assess+178 (60015f26a40, 8000, 10, 0, 0, 10) %l0-3: 0000000000010000 0000060015c05120 0000000000000000 0000000000000032 %l4-7: 0000060015c02888 0000000000000002 0000000000000000 0000060015c05118 000002a100b69a00 genunix:taskq_thread+1a4 (60015c05150, 60015c050f8, 50001, 5751b2acf88, 2a100b69aca, 2a100b69ac8) %l0-3: 0000000000010000 0000060015c05120 0000060015c05128 0000060015c0512a %l4-7: 0000060015c02888 0000000000000002 0000000000000000 0000060015c05118 syncing file systems... 6 done dumping to /dev/dsk/c0t0d0s1, offset...

...D 353471 kern.notice] d1fafd08 genunix:segmap_getmapflt+6fc (fe c23310, d1e897c0,) Jun 13 19:33:04 mathrock-opensolaris genunix: [ID 353471 kern.notice] d1fafd78 lofi:lofi_strategy_task+2c8 (d2b 6bee0, 0, 0, 0, ) Jun 13 19:33:04 mathrock-opensolaris genunix: [ID 353471 kern.notice] d1fafdc8 genunix:taskq_thread+194 (c5e87f 30, 0) Jun 13 19:33:04 mathrock-opensolaris genunix: [ID 353471 kern.notice] d1fafdd8 unix:thread_start+8 () Jun 13 19:33:04 mathrock-opensolaris unix: [ID 100000 kern.notice] Jun 13 19:33:04 mathrock-opensolaris genunix: [ID 672855 kern.notice] syncing file systems... [/b] This mes...

...ead_done (zio=0x2aaabcfa4ed0) at lib/libzpool/build-kernel/arc.c:1850 #5 0x000000000044fb9f in zio_done (zio=0x2aaabcfa4ed0) at lib/libzpool/build-kernel/zio.c:868 #6 0x00000000004527f0 in zio_vdev_io_assess (zio=0x2aaabcfa4ed0) at lib/libzpool/build-kernel/zio.c:1491 #7 0x0000000000466ecf in taskq_thread (arg=<value optimized out>) at lib/libsolkerncompat/taskq.c:160 #8 0x00002afcd74273ca in start_thread () from /lib/libpthread.so.0 #9 0x00002afcd771555d in clone () from /lib/libc.so.6 #10 0x0000000000000000 in ?? () (gdb) frame 2 #2 0x0000000000454dff in arc_change_state (new_state=0x59...

...00000002 0000000000000006 0000000000000006 000002a1011d3940 zfs:zio_vdev_io_assess+178 (60007432bc0, 8000, 10, 0, 0, 10) %l0-3: 0000000000000002 0000000000000002 0000000000000000 0000000000000006 %l4-7: 0000000000000010 000000022042d79b 0000000000000000 0000056ceb2506fa 000002a1011d3a00 genunix:taskq_thread+1a4 (60001b81808, 60001b817b0, 50001, 56f0b67e471, 2a1011d3aca, 2a1011d3ac8) %l0-3: 0000000000010000 0000060001b817d8 0000060001b817e0 0000060001b817e2 %l4-7: 00000600020f7310 0000000000000002 0000000000000000 0000060001b817d0 syncing file systems... 3 1 done dumping to /dev/md/dsk/d10, offset...

...b90 zfs:vdev_disk_io_done+29 () Sep 13 10:32:56 eve genunix: [ID 655072 kern.notice] ffffff0004577bb0 zfs:vdev_io_done+1d () Sep 13 10:32:56 eve genunix: [ID 655072 kern.notice] ffffff0004577bd0 zfs:zio_vdev_io_done+1b () Sep 13 10:32:56 eve genunix: [ID 655072 kern.notice] ffffff0004577c60 genunix:taskq_thread+1cb () Sep 13 10:32:56 eve genunix: [ID 655072 kern.notice] ffffff0004577c70 unix:thread_start+8 () Sep 13 10:32:56 eve unix: [ID 100000 kern.notice] Sep 13 10:32:56 eve genunix: [ID 672855 kern.notice] syncing file systems... Sep 13 10:32:56 eve genunix: [ID 904073 kern.notice] done Sep 13 10:32...

Today I attempted to upgrade to S10_U2 and migrate some mirrored UFS SVM partitions to ZFS. I used Live Upgrade to migrate from U1 to U2 and that went without a hitch on my SunBlade 2000. And the initial conversion of one side of the UFS mirrors to a ZFS pool and subsequent data migration went fine. However, when I attempted to attach the second side mirrors as a mirror of the ZFS pool, all

..., 0, d51e5d) Nov 19 09:09:34 fugly genunix: [ID 353471 kern.notice] d51e5d64 zfs:vdev_io_done+18 (d53a5600, fe964d5d,) Nov 19 09:09:34 fugly genunix: [ID 353471 kern.notice] d51e5d78 zfs:zio_vdev_io_done+e (d53a5600, 0, 0, 0, ) Nov 19 09:09:34 fugly genunix: [ID 353471 kern.notice] d51e5dc8 genunix:taskq_thread+16c (d45d51f0, 0) Nov 19 09:09:34 fugly genunix: [ID 353471 kern.notice] d51e5dd8 unix:thread_start+8 () Nov 19 09:09:34 fugly unix: [ID 100000 kern.notice] Nov 19 09:09:34 fugly genunix: [ID 672855 kern.notice] syncing file systems... Nov 19 09:09:34 fugly genunix: [ID 904073 kern.notice] done No...

...s:zio_next_stage+11e () May 9 16:47:43 sol genunix: [ID 655072 kern.notice] fffffe8000581ba0 zfs:vdev_mirror_io_done+38c () May 9 16:47:43 sol genunix: [ID 655072 kern.notice] fffffe8000581bc0 zfs:zio_vdev_io_done+2d () May 9 16:47:43 sol genunix: [ID 655072 kern.notice] fffffe8000581c60 genunix:taskq_thread+200 () May 9 16:47:43 sol genunix: [ID 655072 kern.notice] fffffe8000581c70 unix:thread_start+8 () May 9 16:47:43 sol unix: [ID 100000 kern.notice] May 9 16:47:43 sol genunix: [ID 672855 kern.notice] syncing file systems... May 9 16:47:43 sol genunix: [ID 904073 kern.notice] done May 9 16:47:...