On Solaris 10, running ''pstack'' against our JVMs (java version "1.5.0_17") can take as long as 60 seconds to complete. During that time apparently, the JVM is unable to do anything else. This causes applications to crash. So they said to me, "Please write a DTrace pstack." Using the syscall provider, I can print out the PID and TID and then a nice jstack() every time a system call is made by the PID, but that shows only the stack for the LWP that ran the system call. That''s not what ''pstack'' does. I want to stop the process, and then walk through each LWP and generate a stack trace. Can I do this in DTrace? Anyone? Thanks. -- This message posted from opensolaris.org
On Thu, Dec 17, 2009 at 09:16:24AM -0800, GGallagher wrote:> On Solaris 10, running ''pstack'' against our JVMs (java version > "1.5.0_17") can take as long as 60 seconds to complete. During that > time apparently, the JVM is unable to do anything else. This causes > applications to crash. So they said to me, "Please write a DTrace > pstack." > > Using the syscall provider, I can print out the PID and TID and then a > nice jstack() every time a system call is made by the PID, but that > shows only the stack for the LWP that ran the system call. That''s not > what ''pstack'' does. I want to stop the process, and then walk through > each LWP and generate a stack trace. Can I do this in DTrace?No, you can''t. To be exact, you can stop a process (with the stop() destructive action), and you can then get the stack traces for all its threads (using the system() action to invoke pstack!), but not in a way that''s distinguishable from actually using pstack. The problem is the "stop the process" part of "stop the process, and then walk through each LWP and generate a stack trace". The stopping of the process is what makes the JVM stop making progress, which in turn sounds like the reason that your apps are crashing (but why are they so sensitive to timing?). DTrace can only give you a stack trace for threads that cause some probe to fire. If a thread is blocking inside a system call at the time that your D script start running, then DTrace will not be able to give you a stack trace for that thread until that thread wakes up and does something that causes one of your probes to fire. Perhaps a tool could be built that does what pstack does, but without stopping the process, and only for threads that are blocking in the kernel, while using DTrace to get stack traces for active threads (probably using a profile provider probe)? Nico --
Gerald, man jstack or Roll your own: Map < Thread, StackTraceElement[] > tmap = Thread.getAllStackTraces(); Cheers, Joel. Nicolas Williams wrote:> On Thu, Dec 17, 2009 at 09:16:24AM -0800, GGallagher wrote: > >> On Solaris 10, running ''pstack'' against our JVMs (java version >> "1.5.0_17") can take as long as 60 seconds to complete. During that >> time apparently, the JVM is unable to do anything else. This causes >> applications to crash. So they said to me, "Please write a DTrace >> pstack." >> >> Using the syscall provider, I can print out the PID and TID and then a >> nice jstack() every time a system call is made by the PID, but that >> shows only the stack for the LWP that ran the system call. That''s not >> what ''pstack'' does. I want to stop the process, and then walk through >> each LWP and generate a stack trace. Can I do this in DTrace? >> > > No, you can''t. To be exact, you can stop a process (with the stop() > destructive action), and you can then get the stack traces for all its > threads (using the system() action to invoke pstack!), but not in a way > that''s distinguishable from actually using pstack. > > The problem is the "stop the process" part of "stop the process, and > then walk through each LWP and generate a stack trace". The stopping of > the process is what makes the JVM stop making progress, which in turn > sounds like the reason that your apps are crashing (but why are they so > sensitive to timing?). > > DTrace can only give you a stack trace for threads that cause some probe > to fire. If a thread is blocking inside a system call at the time that > your D script start running, then DTrace will not be able to give you a > stack trace for that thread until that thread wakes up and does > something that causes one of your probes to fire. > > Perhaps a tool could be built that does what pstack does, but without > stopping the process, and only for threads that are blocking in the > kernel, while using DTrace to get stack traces for active threads > (probably using a profile provider probe)? > > Nico >