Hi everyone, I am not exactly sure if this is ZFS problem or Java or something else? On a T2000 with latest patch 120011-14, we are not able to kill a Java process, e.g. kill or kill -9 has no affect on the process. I did a lsof on the PID and saw over 200 open files and many are showing the following zfs error. Does this have anything to do with us no being able to kill the process? Java doesn''t know about zfs? Thank you for any help, root at as17:/> lsof -p 22861 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 22861 prod cwd unknown file system type (zfs), v_op: 0x600077c7680 java 22861 prod txt VREG 32,8 74244 128850 /usr/jdk1.5.0_11/bin/java java 22861 prod txt unknown file system type (zfs), v_op: 0x600077c7500 java 22861 prod txt unknown file system type (zfs), v_op: 0x600077c7500 java 22861 prod txt unknown file system type (zfs), v_op: 0x600077c7500 java 22861 prod txt unknown file system type (zfs), v_op: 0x600077c7500 This message posted from opensolaris.org
>Hi everyone, >I am not exactly sure if this is ZFS problem or Java or something else? >On a T2000 with latest patch 120011-14, we are not able to kill a Java process, e.g. kill or kill-9 has no affect on the process. I did a lsof on the PID and saw over 200 open files and many are s howing the following zfs error.>Does this have anything to do with us no being able to kill the process? >Java doesn''t know about zfs?lsof is not a supported Solaris tool and apparently the version you are using does now know about ZFS. I''d suggest using with "pstack" or creating a kernel core dump to see where the process is really stuck. As it does not respond to kill it must be stuck in the kernel someplace. Casper
Thanks for your reply, Before I used lsof, I tried pstack and truss -p but I get the following message: # pstack 22861 pstack: cannot examine 22861: unanticipated system error # truss -p 22861 truss: unanticipated system error: 22861 # pstack -F 22861 pstack: cannot examine 22861: unanticipated system error I do get result with pmap. Is there a dtrace script that I can run for this process? Would that help? I will try the core dump as well. Thanks, This message posted from opensolaris.org
You are looking for mdb. echo ''0t22861::pid2proc |::walk thread |::findstack'' | mdb -k On Tue, Mar 18, 2008 at 11:28 PM, Vahid Moghaddasi <vahid at cckeeper.com> wrote:> Thanks for your reply, > Before I used lsof, I tried pstack and truss -p but I get the following message: > # pstack 22861 > pstack: cannot examine 22861: unanticipated system error > # truss -p 22861 > truss: unanticipated system error: 22861 > # pstack -F 22861 > pstack: cannot examine 22861: unanticipated system error > I do get result with pmap. > > Is there a dtrace script that I can run for this process? Would that help? > I will try the core dump as well. > Thanks, > > > > This message posted from opensolaris.org > _______________________________________________ > > > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >-- Just me, Wire ... Blog: <prstat.blogspot.com>
> You are looking for mdb. > > echo ''0t22861::pid2proc |::walk thread |::findstack'' > | mdb -k >There are over 120 threads in the very long output, I will post some sections of the output here but they are mostly look alike, what can I do with this information, is there a way to kill 22861 at all? Thanks. stack pointer for thread 3000369f920: 2a102618db1 [ 000002a102618db1 stop+0x8c4() ] 000002a102618e61 issig_forreal+0x1e4() 000002a102618f91 cv_timedwait_sig+0x204() 000002a102619041 cv_waituntil_sig+0x8c() 000002a102619111 poll_common+0x164() 000002a102619201 pollsys+0xf8() 000002a1026192e1 syscall_trap32+0xcc() stack pointer for thread 30060700060: 2a106940f61 [ 000002a106940f61 stop+0x8c4() ] 000002a106941011 issig_forreal+0x1e4() 000002a106941141 lwp_cond_wait+0x550() 000002a1069412e1 syscall_trap32+0xcc() ... ... stack pointer for thread 30005420d00: 2a1052e7ec1 [ 000002a1052e7ec1 cv_wait+0x38() ] 000002a1052e7f71 page_lock_es+0x214() 000002a1052e8021 pvn_vplist_dirty+0x2f8() 000002a1052e8101 nfs_putpages+0x124() 000002a1052e81c1 nfs3_putpage+0xcc() 000002a1052e8271 fop_putpage+0x1c() 000002a1052e8321 nfs_purge_caches+0xe4() 000002a1052e83d1 nfs_attr_cache+0x270() 000002a1052e8481 nfs3_getattr_otw+0x1b8() 000002a1052e85f1 nfs3_validate_caches+0x4c() 000002a1052e8731 nfs3_getpage+0xa4() 000002a1052e8861 fop_getpage+0x44() 000002a1052e8931 segmap_getmapflt+0x588() 000002a1052e8a41 snf_segmap+0x13c() 000002a1052e8bc1 sosendfile64+0x2a8() 000002a1052e8d21 sendvec64+0xd4() 000002a1052e8f61 sendfilev+0x178() 000002a1052e92e1 syscall_trap32+0xcc() ... stack pointer for thread 30004ec6d00: 2a102dc0f61 [ 000002a102dc0f61 stop+0x8c4() ] 000002a102dc1011 issig_forreal+0x1e4() 000002a102dc1141 lwp_cond_wait+0x550() 000002a102dc12e1 syscall_trap32+0xcc() stack pointer for thread 300537f6fa0: 2a1034c8f61 [ 000002a1034c8f61 stop+0x8c4() ] 000002a1034c9011 issig_forreal+0x1e4() 000002a1034c9141 lwp_cond_wait+0x550() 000002a1034c92e1 syscall_trap32+0xcc() This message posted from opensolaris.org