dtrace discuss - Nov 2006 - remembering last 5 syscalls per PID?

Hello,

 I have a problem with some FastCGI processes. Sometimes, they hang up in kernel
space (probably in a syscall) and I can''t figure out which syscall it
is. Running truss on the process reveals no information, since it hangs there.

 I need some help writing DTrace script, which would remember last five syscalls
(with arguments) per PID and when I hit ctrl-c, it displays these information.

 Unfortunately, I don''t know PID in advance, nor any other information
(there are lots of FastCGI children on that machine). I also need something,
that is not performance-intensive. I''ve written some DTrace scripts
before, but doing this, I have several problems:

- how do I get the string arguments of miscellaneous syscall:::entry or
syscall:::return probes? I can''t get args[0] for unspecified call, but
I would like to know filename. All "truss-simulating" DTrace scripts
I''ve seen don''t actually display strings (they use arg0).
- how do I save "last 5 entries per PID". I''ve been thinking
about simple array (syscalls[pid,0], ..., syscalls[pid,1]), but I don''t
know how to write array (since D does not have cycles). Aggregations
don''t seem to work for this either, since I need to have string values.

Any ideas are welcome. Thank you.
 
 
This message posted from opensolaris.org

On 11/6/06, Juraj Bednar <jooray at gmail.com> wrote:
>  I have a problem with some FastCGI processes. Sometimes, they hang up in
kernel > space (probably in a syscall) and I can''t figure out which
syscall it is. Running truss
> on the process reveals no information, since it hangs there.
>
>  I need some help writing DTrace script, which would remember last five
syscalls (with > arguments) per PID and when I hit ctrl-c, it displays these
information.
Any reason you can''t use pstack to get the call stack when the process
hangs?

Thanks,
- Ryan
-- 
UNIX Administrator
http://prefetch.net

Hi Juraj,

Juraj Bednar wrote:
> Hello,
> 
> I have a problem with some FastCGI processes. Sometimes, they hang up
> in kernel space (probably in a syscall) and I can''t figure out
which
> syscall it is. Running truss on the process reveals no information,
> since it hangs there.
> 
> I need some help writing DTrace script, which would remember last
> five syscalls (with arguments) per PID and when I hit ctrl-c, it
> displays these information.
While this task may be interesting exercise for learning DTrace, I think 
there''s a simpler way to get an idea where it is in kernel. You may try
something like this (replace <pid> with actual PID of your process):

echo "0t<pid>::pid2proc | ::walk thread | ::findstack" | mdb -k



Hth,
Victor