dtrace discuss - Aug 2006 - Using pid provider when process forks

I''m trying to do some userlevel tracing with the pid provider[1]. 
I''ve
got two static probes set up, and I want to trace all the user functions 
that the program calls between those two probes.

The problem is, I won''t know in advance what the PID of the process I 
want to trace is, since it will be the child (or quite possibly 
grandchild, or even great-grandchild) of another process.

So I tried this:

sendmail*:::queue-createentry-start
/zonename == $1/
{
     interesting_pid = pid;
}

sendmail*:::queue-createentry-end
/interesting_pid && zonename == $1/
{
     interesting_pid = 0;
}

pid*:::entry
/interesting_pid == pid && zonename == $1/
{
     printf("%s\n", probefunc);
}

That refuses to run.  dtrace(1) tells me:

   dtrace: failed to compile script ./queue-create-pid.d: line 27:
     probe description pid*:::entry does not match any probes

That sort of makes sense and sort of doesn''t.  I understand that pid 
probes are created on the fly, but I thought ''*'' was supposed
to match
everything.  Of course, that could have a huge overhead.

I tried replacing ''pid*'' with ''syscall'', to
make sure that the overall
logic of the script was correct, and that does work.  Except, of course, 
it prints system call names, rather than function names.

Doing some digging, it looks like I''ve bumped in to a variant of this:

     http://www.opensolaris.org/jive/thread.jspa?messageID=33783

but I thought I''d double check, since DTrace development seems to
happen
quite quickly.

Assuming that the approach I''ve got above isn''t going to work,
is there
a way that I can achieve this goal with DTrace?

Thanks,

N

[1] to see if I can find out where my missing nanoseconds have gone.

Nik Clayton stated:
< I''m trying to do some userlevel tracing with the pid provider[1]. 
I''ve
< got two static probes set up, and I want to trace all the user functions 
< that the program calls between those two probes.
< 
< The problem is, I won''t know in advance what the PID of the
process I
< want to trace is, since it will be the child (or quite possibly 
< grandchild, or even great-grandchild) of another process.
< 
< So I tried this:
< 
< sendmail*:::queue-createentry-start
< /zonename == $1/
< {
<     interesting_pid = pid;
< }
< 
< sendmail*:::queue-createentry-end
< /interesting_pid && zonename == $1/
< {
<     interesting_pid = 0;
< }
< 
< pid*:::entry
< /interesting_pid == pid && zonename == $1/
< {
<     printf("%s\n", probefunc);
< }
< 
< That refuses to run.  dtrace(1) tells me:
< 
<   dtrace: failed to compile script ./queue-create-pid.d: line 27:
<     probe description pid*:::entry does not match any probes
< 
< That sort of makes sense and sort of doesn''t.  I understand that
pid
< probes are created on the fly, but I thought ''*'' was
supposed to match
< everything.  Of course, that could have a huge overhead.
< 
< I tried replacing ''pid*'' with
''syscall'', to make sure that the overall
< logic of the script was correct, and that does work.  Except, of course, 
< it prints system call names, rather than function names.

  You could spawn a second dtrace script once a first one has caught the
  sendmail*:::queue-createentry-start.

  I''ve used this technique before when dtracing daemons that fork kids
instead
  of spawning threads.

  Something like:

# first dtrace script

sendmail*:::queue-createentry-start
/zonename == $1/
{
    interesting_pid = pid;
    stop(pid);
    spawn("dtrace -s /second/dtrace/script.d -p %d %s", pid,
zonename);
}

# second dtrace script

BEGIN
{
        spawn("prun $target");
}

sendmail*:::queue-createentry-end
/pid == $target && zonename == $$1/
{
    exit(0);
}

pidi$target:::entry
/zonename == $$1/
{
    printf("%s\n", probefunc);
}

# end scripts


Regards,
Sean.
.
< 
< Doing some digging, it looks like I''ve bumped in to a variant of
this:
< 
<     http://www.opensolaris.org/jive/thread.jspa?messageID=33783
< 
< but I thought I''d double check, since DTrace development seems to
happen
< quite quickly.
< 
< Assuming that the approach I''ve got above isn''t going to
work, is there
< a way that I can achieve this goal with DTrace?
< 
< Thanks,
< 
< N
< 
< [1] to see if I can find out where my missing nanoseconds have gone.
< _______________________________________________
< dtrace-discuss mailing list
< dtrace-discuss at opensolaris.org

Sean McGrath - Sun Microsystems Ireland wrote:
>   I''ve used this technique before when dtracing daemons that fork
kids instead
>   of spawning threads.
Now that is a nice little trick.  Thanks for that.

N

On Wed, Aug 09, 2006 at 05:29:25PM +0100, Sean McGrath - Sun Microsystems
Ireland wrote:
>   You could spawn a second dtrace script once a first one has caught the
>   sendmail*:::queue-createentry-start.
> 
>   I''ve used this technique before when dtracing daemons that fork
kids instead
>   of spawning threads.
Yup, basically stop() the new child in proc:::create, uses system() to
start a new dtrace command to trace the child and use system in a BEGIN
probe in the new dtrace command to prun the child.

You can also use this sort of trick to trace, say, exec(2)s of certain
programs.

I''d like to be able to do this without fork()/exec()ing a new dtrace
consumer and prun(1) (with two new actions, say, run(), and dtrace()),
and perhaps that could be done with the Java DTrace API, but it''s not
so
bad.

Nico
--

Sean,

When you wrote this, was it off the top of your head, or are there new 
DTrace features in the works?

The reason I ask is:
Sean McGrath - Sun Microsystems Ireland wrote:
> sendmail*:::queue-createentry-start
> /zonename == $1/
> {
>     interesting_pid = pid;
>     stop(pid);
>     spawn("dtrace -s /second/dtrace/script.d -p %d %s", pid,
zonename);
> }
system(), in my copy of DTrace (SE b40).  Is spawn() a new call?
> # second dtrace script
> 
> BEGIN
> {
>         spawn("prun $target");
> }
system("prun %d", $target);

As ''$target'' doesn''t interpolate in double quoted
strings.

Anyway, with those replacements it works a treat, and I''ve got
something
a bit more general now.

One thought -- given that DTrace can stop a process, it seems sensible 
that there should be an equivalent resume(), or continue() call, that 
does the equivalent of

     system("prun %d", pid_to_run);

That saves the overhead of having to start a whole new process, and 
would be more portable to operating systems that have prun 
functionality, but don''t call it prun (e.g., there''s no prun
on FreeBSD,
and I don''t think OS X has it either).

N

Nik Clayton stated:
< Sean,
< 
< When you wrote this, was it off the top of your head, or are there new 
< DTrace features in the works?

   Ya twas off somewhere from my head.. :)

< 
< The reason I ask is:
< Sean McGrath - Sun Microsystems Ireland wrote:
< >sendmail*:::queue-createentry-start
< >/zonename == $1/
< >{
< >    interesting_pid = pid;
< >    stop(pid);
< >    spawn("dtrace -s /second/dtrace/script.d -p %d %s", pid,
zonename);
< >}
< 
< system(), in my copy of DTrace (SE b40).  Is spawn() a new call?

   You''re right, its system.  Theres no dtrace spawn() function...

< 
< ># second dtrace script
< >
< >BEGIN
< >{
< >        spawn("prun $target");
< >}
< 
< system("prun %d", $target);
< 
< As ''$target'' doesn''t interpolate in double
quoted strings.

  Right again (now I''m feeling the shame of all my typos :)

< 
< Anyway, with those replacements it works a treat, and I''ve got
something
< a bit more general now.

  Cool, glad to hear it.

< 
< One thought -- given that DTrace can stop a process, it seems sensible 
< that there should be an equivalent resume(), or continue() call, that 
< does the equivalent of
< 
<     system("prun %d", pid_to_run);
< 
< That saves the overhead of having to start a whole new process, and 
< would be more portable to operating systems that have prun 
< functionality, but don''t call it prun (e.g., there''s no
prun on FreeBSD,
< and I don''t think OS X has it either).

  Sending a stopped process a SIGCONT should allow stopped process to continue.
  This should be doable from FreeBSD or OS-X right ?

  Thats basically what prun does:
http://cvs.opensolaris.org/source/xref/on/usr/src/cmd/ptools/prun/prun.c

  Though it doesn it nicer, by grabbing the proc/lwp beforehand and some
  cleaning up after sending a SIGCONT.

  
< 
< N

-- 
Sean.
.

On Thu, Aug 10, 2006 at 08:56:01AM +0100, Nik Clayton
wrote:
> One thought -- given that DTrace can stop a process, it seems sensible 
> that there should be an equivalent resume(), or continue() call, that 
> does the equivalent of
> 
>     system("prun %d", pid_to_run);
> 
> That saves the overhead of having to start a whole new process, and 
> would be more portable to operating systems that have prun 
> functionality, but don''t call it prun (e.g., there''s no
prun on FreeBSD,
> and I don''t think OS X has it either).
This discusion has come up before, and the reason we don''t have a
continue()
or resume() action is that it would necessarily not mirror the stop() action
as the stop() action affects the currently executing process. As the home-
spun wisdom goes: stopped processes fire no probes.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

On Thu, Aug 10, 2006 at 08:30:34AM -0700, Adam Leventhal
wrote:
> On Thu, Aug 10, 2006 at 08:56:01AM +0100, Nik Clayton wrote:
> > One thought -- given that DTrace can stop a process, it seems sensible
> > that there should be an equivalent resume(), or continue() call, that 
> > does the equivalent of
> > 
> >     system("prun %d", pid_to_run);
> > 
> > That saves the overhead of having to start a whole new process, and 
> > would be more portable to operating systems that have prun 
> > functionality, but don''t call it prun (e.g., there''s
no prun on FreeBSD,
> > and I don''t think OS X has it either).
> 
> This discusion has come up before, and the reason we don''t have a
continue()
> or resume() action is that it would necessarily not mirror the stop()
action
> as the stop() action affects the currently executing process. As the home-
> spun wisdom goes: stopped processes fire no probes.
But surely the BEGIN probe does fire, even when the target is stopped.

So if you''re starting a new dtrace command to trace a stopped process
then a resume() action would avoid one more round of forking and
execing.

A built-in dtrace() action that could load a new script with a new
target but in the same dtrace consumer where it''s processed would also
avoid a round of fork/exec.

How important is this?  I dunno.  If you''re trying to trace a
production
application that fork()s and exec()s then the delay to do several
fork()/exec()s might not be acceptable.  But most of the time I think
one can very well live with it.

Nico
--

Adam Leventhal wrote:
> On Thu, Aug 10, 2006 at 08:56:01AM +0100, Nik Clayton wrote:
>> One thought -- given that DTrace can stop a process, it seems sensible 
>> that there should be an equivalent resume(), or continue() call, that 
>> does the equivalent of
>>
>>     system("prun %d", pid_to_run);
>>
>> That saves the overhead of having to start a whole new process, and 
>> would be more portable to operating systems that have prun 
>> functionality, but don''t call it prun (e.g., there''s
no prun on FreeBSD,
>> and I don''t think OS X has it either).
> 
> This discusion has come up before, and the reason we don''t have a
continue()
> or resume() action is that it would necessarily not mirror the stop()
action
> as the stop() action affects the currently executing process. As the home-
> spun wisdom goes: stopped processes fire no probes.
Right, but this would be from another process.  So it wouldn''t be 
resume(void), it would be resume(pid_t).  And presumably the BEGIN probe 
fires -- it must do, otherwise

   BEGIN
   {
       system("prun %d", $1);
   }

wouldn''t work.  It would be nice to be able to do:

   BEGIN
   {
       resume($1);
   }

instead.

N

> Right, but this would be from another process.  So it wouldn''t be 
> resume(void), it would be resume(pid_t).  And presumably the BEGIN probe 
> fires -- it must do, otherwise
> 
>   BEGIN
>   {
>       system("prun %d", $1);
>   }
> 
> wouldn''t work.  It would be nice to be able to do:
> 
>   BEGIN
>   {
>       resume($1);
>   }
> 
> instead.
Agreed that it would be nice -- I think the concern is that it would also
be a tad misleading:  stop() occurs in probe context, and guarantees that
the stopped process will not return to user-level, but resume() would
necessarily be a library action that would occur in postprocessing at
userlevel.  That is, the latency between resume() being called and the
target process _actually_ being resumed will be dictated by the buffer
switch rate.  Using system() is a bit clunky, but it''s also clearer
about
what''s actually going on and what the expectations should be.  (Which
is
not to say that adding resume() wouldn''t be an improvement, just that
it''s
not a no-brainer.)

Of course, the real RFE here is following on fork.  Unfortunately, it''s
hard -- there are lots of nasty little semantics issues, to say nothing of
the non-trivial mechanical issues to get it working.  And worse, the
stop()/system("prun") trick is Good Enough for many (myself included,
having
used this for the zillionth time to nail a problem just yesterday).  So
to be honest (and as much as it pains me to say it), we''re unlikely to
solve
this in the near-term future...

Finally (and for whatever it''s worth), note that this trick is outlined
on
slide 28 of our Tips, Tricks and Gotchas presentation:

  http://blogs.sun.com/roller/resources/bmc/dtrace_tips.pdf

That presentation is packed with information not readily available elsewhere;
advanced and avid DTrace users are strongly recommended to go through it 
periodically...

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

Nik Clayton wrote:
> I''m trying to do some userlevel tracing with the pid provider[1]. 
I''ve
> got two static probes set up, and I want to trace all the user functions 
> that the program calls between those two probes.
[...]
> Assuming that the approach I''ve got above isn''t going to
work, is there
> a way that I can achieve this goal with DTrace?
Thanks for all the useful replies, esp. Sean McGrath, not only first to 
respond, but with code too.

I''ve written this technique up at:

http://jc.ngo.org.uk/blog/2006/08/10/day-32-of-60-complete-instrumentation-of-queue-creation/

in the hope that others might find it useful too.

N