dtrace discuss - Apr 2008 - Help pid provider follow child processes

I''ve just recently started working with DTrace and one of the first  
issues I''ve run across is the difficulty in following child processes  
when instrumenting user space. I''ve seen some scraps here and there  
about how to use the pid provider together with the proc or syscall  
provider to pstop, instrument, and pstart child processes, but there  
are some additional cautionary notes indicating that this is slightly  
more complex than it might appear at first blush. (Namely: the rtld  
has to complete some degree of initial linking to enable library- 
qualified probes properly, and the mechanics haven''t been explained  
adequately that I feel this is something I can work out for myself.  
Has anyone been able to do this back-to-front such that they can  
provide something of a recipe for how to do this? I realise that this  
is somewhat un-DTrace like at some basic level, at least with this  
kind of implementation. If the approach is misconceived and another  
alternative available, I''m happy to be pointed in the right direction.)

Second: I''m a bit confused about how to grab userland data into dtrace
buffers, with or without pointers. There''s a rather jumbled example on
the "Gotcha: misuse of copyin" slide from the "Advanced
DTrace"
presentation
(http://kr.sun.com/developers/solaris/techdocs/dtrace_tips_public.pdf
  -- there''s some basic syntax errors here, minimally at the level of  
some missing grouping symbols, that I''ve tried unsuccessfully to clean
up), and the other examples of dereferencing from the docs (the  
chapter from the manual on documenting user apps that references  
write(), the example for uname() from
http://learningsolaris.com/docs/dtrace_course.2005.8.18.pdf)
. I tried something very much like Adam''s uname() example to try to  
instrument calls to getservyname_r() and getservbyport_r(), but  
something of it has eluded me and a far more skilled colleague. I have  
a sinking suspicion that this is far simpler than I''ve flattered  
myself into thinking it''s not.

Cheers,
Bayard

On Mon, 2008-04-14 at 23:03 +0100, Bayard Bell wrote:
> I''ve just recently started working with DTrace and one of the
first
> issues I''ve run across is the difficulty in following child
processes
> when instrumenting user space.
Are trying to survive fork/exec or just a single process forking its
image?
> Second: I''m a bit confused about how to grab userland data into
dtrace
> buffers, with or without pointers. There''s a rather jumbled
example on
> the "Gotcha: misuse of copyin" slide from the "Advanced
DTrace"
> presentation
(http://kr.sun.com/developers/solaris/techdocs/dtrace_tips_public.pdf
>   -- there''s some basic syntax errors here, minimally at the level
of
> some missing grouping symbols, that I''ve tried unsuccessfully to
clean
> up), and the other examples of dereferencing from the docs (the  
> chapter from the manual on documenting user apps that references  
> write(), the example for uname() from
http://learningsolaris.com/docs/dtrace_course.2005.8.18.pdf)
> . I tried something very much like Adam''s uname() example to try
to
> instrument calls to getservyname_r() and getservbyport_r(), but  
> something of it has eluded me and a far more skilled colleague. I have  
> a sinking suspicion that this is far simpler than I''ve flattered  
> myself into thinking it''s not.
I''m not sure I really understand what seems to be a problem here.

Thanks,
Roman.

On 16 Apr 2008, at 23:32, Roman Shaposhnik wrote:
> On Mon, 2008-04-14 at 23:03 +0100, Bayard Bell wrote:
>> I''ve just recently started working with DTrace and one of the
first
>> issues I''ve run across is the difficulty in following child
processes
>> when instrumenting user space.
>
> Are trying to survive fork/exec or just a single process forking its
> image?
I''m looking to re-initialise the same set of probes under the pid  
provider against child processes subsequently created by fork/exec.
>> Second: I''m a bit confused about how to grab userland data
into
>> dtrace
>> buffers, with or without pointers. There''s a rather jumbled
example
>> on
>> the "Gotcha: misuse of copyin" slide from the "Advanced
DTrace"
>> presentation
(http://kr.sun.com/developers/solaris/techdocs/dtrace_tips_public.pdf
>>  -- there''s some basic syntax errors here, minimally at the
level of
>> some missing grouping symbols, that I''ve tried unsuccessfully
to
>> clean
>> up), and the other examples of dereferencing from the docs (the
>> chapter from the manual on documenting user apps that references
>> write(), the example for uname() from
http://learningsolaris.com/docs/dtrace_course.2005.8.18.pdf)
>> . I tried something very much like Adam''s uname() example to
try to
>> instrument calls to getservyname_r() and getservbyport_r(), but
>> something of it has eluded me and a far more skilled colleague. I  
>> have
>> a sinking suspicion that this is far simpler than I''ve
flattered
>> myself into thinking it''s not.
>
> I''m not sure I really understand what seems to be a problem here.
Maybe I''m just picking up an example that''s sufficiently
disparate
from others I think I understand, such that I''m so confused about the  
differences that I''m not able to account for these with code.  My  
colleague posted sample code for the problem here:

http://forum.java.sun.com/thread.jspa?threadID=5285918&tstart=0

I''ve tried a couple of variations on this and suspect that I''m
missing
something basic.
> Thanks,
> Roman.
>

On Apr 16, 2008, at 4:00 PM, Bayard Bell wrote:
>
> On 16 Apr 2008, at 23:32, Roman Shaposhnik wrote:
>
>> On Mon, 2008-04-14 at 23:03 +0100, Bayard Bell wrote:
>>> I''ve just recently started working with DTrace and one of
the first
>>> issues I''ve run across is the difficulty in following
child
>>> processes
>>> when instrumenting user space.
>>
>> Are trying to survive fork/exec or just a single process forking its
>> image?
>
> I''m looking to re-initialise the same set of probes under the pid
> provider against child processes subsequently created by fork/exec.
Well, if you''re really talking about re-inserting probes into a  
completely new image (after the exec)
than I consider DTrace to be doing exactly the right thing -- these  
are brand new probes so they
have to be reinserted.
> Maybe I''m just picking up an example that''s sufficiently
disparate
> from others I think I understand, such that I''m so confused about
> the differences that I''m not able to account for these with code.
> My colleague posted sample code for the problem here:
>
> http://forum.java.sun.com/thread.jspa?threadID=5285918&tstart=0
>
> I''ve tried a couple of variations on this and suspect that
I''m
> missing something basic.
Indeed. You''re missing -32 option to /usr/sbin/dtrace. See my reply on
the forum.

Thanks,
Roman.

>>
>> On 16 Apr 2008, at 23:32, Roman Shaposhnik wrote:
>>
>>> On Mon, 2008-04-14 at 23:03 +0100, Bayard Bell wrote:
>>>> I''ve just recently started working with DTrace and one
of the first
>>>> issues I''ve run across is the difficulty in following
child
>>>> processes
>>>> when instrumenting user space.
>>>
>>> Are trying to survive fork/exec or just a single process forking
its
>>> image?
>>
>> I''m looking to re-initialise the same set of probes under the
pid
>> provider against child processes subsequently created by fork/exec.
> Well, if you''re really talking about re-inserting probes into a  
> completely new image (after the exec)
> than I consider DTrace to be doing exactly the right thing -- these  
> are brand new probes so they have to be reinserted.
I am dealing with same problem as Bayard trying to trace child processes.
>From Roman''s answer I do not exactly know if it is possible or not.
What do you
mean with "....DTrace to be doing exactly the right thing -- these  are
brand new
probes so they have to be reinserted....."?? I''ve tried to trace
my test program which spawns
child and I really wasn''t able to see any trace output for that child
from pid provider.
>From my point of view as the name of probe "pidPID" has to be
specified
statically I can see no way how to trace child processes (as we don''t
know
PID when starting the script) without destructive hacks like ones described in
this thread
http://mail.opensolaris.org/pipermail/dtrace-discuss/2007-April/003733.html.

I consider the way how pid provider works a big drawback of dtrace for user-land
debugging/tracing. I often need to debug complex multi-process
application/service
 where I just can''t do something like "dtrace -c app".
Cosider a daemon - after start,
daemons often spawns new process, daemonize  child with setsid etc. and letting 
parent process die. Yes I can look for PID after that and run new dtrace session
but
then I will lose lot of interesting information (like deamon initialization). 
Other good examples are short-lived processes spawned by master process - you 
really do need to know what''s happening inside them. I would like to
see something
like shortlived.d script from DTraceToolkit but with capability to trace
function calls not
only time spent in them.

Other thing that would be possible if there wouldn''t be a need to
specify PID statically
within pid provider  would be nice library tracing. Wouldn''t be nice if
following would be possible?:

:mylibrary.so:myfunction:entry {
 printf("myfunction: parameter x %d was supplied by program %s with PID
%d\n",arg[0],execname,pid);
}

With this I could as questions like:
  What programs are using this library function?
  What program uses this library function most frequently? 
  What is the common pattern that this function is receiving?

However I really do not know if it is technically possible to create
"dynamic pid provider" :-)

J.

On Wed, 2008-04-30 at 17:51 +0200, broker5 at seznam.cz
wrote:
> > Well, if you''re really talking about re-inserting probes into
a
> > completely new image (after the exec)
> > than I consider DTrace to be doing exactly the right thing -- these  
> > are brand new probes so they have to be reinserted.
> 
> I am dealing with same problem as Bayard trying to trace child processes. 
> >From Roman''s answer I do not exactly know if it is possible or
not.
It is possible via the system action that would reinsert the probes
into the brand new PID.
> What do you 
> mean with "....DTrace to be doing exactly the right thing -- these 
are brand new
> probes so they have to be reinserted....."?? I''ve tried to
trace my test program which spawns
> child and I really wasn''t able to see any trace output for that
child from pid provider.
That''s why I asked my question -- I can agree that if the process
simply
forks and does NOT do an exec DTrace can be expected (although not
required) to keep the probes around. Once exec is done -- all bets are
off anyway. You have to reinsert the probes. Regardless of whether
you do it manually (via system("dtrace...");) or you have your
dynamic PID provider. Works needs to be done. Unlike with pure forks
where in fact, work is done to *remove* probes from the newborn child.
> However I really do not know if it is technically possible to create
"dynamic pid provider" :-)
Well, start hacking, then ;-)

Thanks,
Roman.

Hi Roman,
> On Wed, 2008-04-30 at 17:51 +0200, broker5 at seznam.cz wrote:
> > > Well, if you''re really talking about re-inserting probes
into a
> > > completely new image (after the exec)
> > > than I consider DTrace to be doing exactly the right thing --
these
> > > are brand new probes so they have to be reinserted.
> > 
> > I am dealing with same problem as Bayard trying to trace child
processes.
> > >From Roman''s answer I do not exactly know if it is
possible or not.
> 
> It is possible via the system action that would reinsert the probes
> into the brand new PID.
I tried to use system() to reinsert probes using proc:::exec-success or
proc:::start but
execution of the process in which context it was fired is not synchronized with
new dtrace
instance started in system(). The result is that with this I can''t
trace short-lived
processes because they are finished before new dtrace instance in system()
reinserts
probes (it takes some time). 

Do you know some workaround ?

Thanks J.

broker5 at seznam.cz wrote:
> Hi Roman,
> 
>> On Wed, 2008-04-30 at 17:51 +0200, broker5 at seznam.cz wrote:
>>>> Well, if you''re really talking about re-inserting
probes into a
>>>> completely new image (after the exec)
>>>> than I consider DTrace to be doing exactly the right thing --
these
>>>> are brand new probes so they have to be reinserted.
>>> I am dealing with same problem as Bayard trying to trace child
processes.
>>> >From Roman''s answer I do not exactly know if it is
possible or not.
>> It is possible via the system action that would reinsert the probes
>> into the brand new PID.
> 
> I tried to use system() to reinsert probes using proc:::exec-success or
proc:::start but
> execution of the process in which context it was fired is not synchronized
with new dtrace
> instance started in system(). The result is that with this I can''t
trace short-lived
> processes because they are finished before new dtrace instance in system()
reinserts
> probes (it takes some time). 
> 
> Do you know some workaround ?
the usual thing to do is to stop() the process in question and then do 
the system() thing.

HTH
Michael
-- 
Michael Schuster 	http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''

On Wed, Apr 30, 2008 at 05:51:15PM +0200, broker5 at seznam.cz
wrote:
> I am dealing with same problem as Bayard trying to trace child processes. 
> [...]
> However I really do not know if it is technically possible to create
"dynamic pid provider" :-)
The way I''ve done this in the past is to use syscall or proc probes to
stop() the child process, followed by a system() action to start a new
instance of DTrace to trace the new process.

I don''t see how to create a "dynamic pid provider" without
stopping the
new child process to give the pid provider a chance to get a proc handle
to it and do all that it has to do to setup pid provider probes.  Which
means that a "dynamic pid provider" would have to be destructive.

But, yeah, it''d be nice to not have to system() a new DTrace instance.
In Java I think you could certainly do it by using the Java interfaces
to DTrace.  But if you''re using dtrace(1M) then yes, you have to start
a
new instance.

Nico
--

> ----------------------------------------
> On Wed, Apr 30, 2008 at 05:51:15PM +0200, broker5 at seznam.cz wrote:
> > I am dealing with same problem as Bayard trying to trace child
processes.
> > [...]
> > However I really do not know if it is technically possible to create
"dynamic
> pid provider" :-) 
> 
> The way I''ve done this in the past is to use syscall or proc
probes to
> stop() the child process, followed by a system() action to start a new
> instance of DTrace to trace the new process.
> 
> I don''t see how to create a "dynamic pid provider"
without stopping the
> new child process to give the pid provider a chance to get a proc handle
> to it and do all that it has to do to setup pid provider probes.  Which
> means that a "dynamic pid provider" would have to be destructive.
> 
> But, yeah, it''d be nice to not have to system() a new DTrace
instance.
> In Java I think you could certainly do it by using the Java interfaces
> to DTrace.  But if you''re using dtrace(1M) then yes, you have to
start a
> new instance.
> 
> Nico

I just tried stop() as you suggested and I thing something bad is happening
when you are using this action. Please see following. Having this two very 
simple dtrace scripts:

#!/usr/sbin/dtrace -wqs
/*this is master.d dscript file*/

proc:::exec-success
/(execname == "myprog") || its_ours[ppid]/
{
    stop(); /*try to comment this line*/
    its_ours[pid] = 1;
    printf("proccess is %s, %d, %d\n",execname,pid,ppid);
    system("dtrace -s helper.d %d > _%s_%d
&",pid,execname,pid);
}

------------------EOF
and:

/*this is helper.d dscript file*/
#pragma D option destructive
#pragma D option quiet

dtrace:::BEGIN
{
    system("prun %d", $1); /*try to comment this line*/
}

pid$1:::entry
{
  self->indent++;
  printf("%*s--> %s(%d)\n",2*self->indent,"
",probefunc,tid);
}

pid$1:::return
{
  printf("%*s<-- %s(%d)\n",2*self->indent,"
",probefunc,tid);
  self->indent = (self->indent-1 < 0 ? 0 : self->indent-1);
}

----------------------EOF

Here is my testing C program "myprog": 

void
fce() {
  sleep(1);
  write(1,"W",1);
}

int main() {
  while(1) fce();

---------------------EOF

Having this I first run master.d script and then I run myprog. 
Now it seems that if you are using stop() and prune then probes from
shared libraries are not fired so you will not see execution flow inside any 
shared library. If you are not using stop() then everything works ok. You can 
see output similar to this if you will leave stop() and prune line in my scripts
commented:

  --> fce(1)
    --> sleep(1)
      --> _save_nv_regs(1)
      <-- _save_nv_regs(1)
      --> ___nanosleep(1)
      <-- ___nanosleep(1)
    <-- sleep(1)
    --> write(1)
      --> _save_nv_regs(1)
      <-- _save_nv_regs(1)
      --> _write(1)
      <-- _write(1)
    <-- write(1)
  <-- fce(1)

But if you will uncomment that lines, you will see something like this (at the
end of
output file after lot of initialization lines):

            <-- fce(1)
            --> fce(1)
            <-- fce(1)
            --> fce(1)
            <-- fce(1)
            --> fce(1)
            <-- fce(1)
            --> fce(1)

Is this some sort of bug or am I doing something wrong? Could someone try above
procedure on your computer, maybe something is broken on mine ?

Thanks J.

> I just tried stop() as you suggested and I thing something bad is happening
> when you are using this action. Please see following. Having this two very 
> simple dtrace scripts:
> 
> #!/usr/sbin/dtrace -wqs
> /*this is master.d dscript file*/
> 
> proc:::exec-success
My unqualified guess is that exec-success probe is executed right after
the process image is replaced by new one, which is way before any
symbols are loaded from any libraries. So setting in the second script
pid$1:::entry sets probes only in symbols defined in the binary itself.

I tried to create another scenario with third dtrace script in the
middle between master and helper and it seems to work as requested.

I''m attaching the scripts.

On the side note, if you don''t want to see realtime linker in the
output, you can try to run the binary with LD_BIND_NOW environment set
(see man ld.so.1)

Hope this helps

-- 
	Vlad
-------------- next part --------------
#!/usr/sbin/dtrace -wqs
/*this is master.d dscript file*/

proc:::exec-success
/(execname == "myprog") || its_ours[ppid]/
{
    stop(); /*try to comment this line*/
    its_ours[pid] = 1;
    printf("proccess is %s, %d, %d\n",execname,pid,ppid);
    system("./middle.d %d &",pid);
}
-------------- next part --------------
#pragma D option destructive
#pragma D option quiet

dtrace:::BEGIN
{
    system("prun %d", $1); /*try to comment this line*/
}

pid$1:::entry
{
  self->indent++;
  printf("%*s--> %s(%d)\n",2*self->indent,"
",probefunc,tid);
}

pid$1:::return
{
  printf("%*s<-- %s(%d)\n",2*self->indent,"
",probefunc,tid);
  self->indent = (self->indent-1 < 0 ? 0 : self->indent-1);
}
-------------- next part --------------
#!/usr/sbin/dtrace -s
#pragma D option destructive
#pragma D option quiet

dtrace:::BEGIN
{
	system("prun %d", $1); /*try to comment this line*/
	printf ("pid %d armed\n", pid);
}

pid$1::main:entry
{
	stop();
	system("dtrace -s helper.d %d > _%s_%d &",pid,execname,pid);
	exit(0);
}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 193 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080507/e1bf4ea6/attachment.bin>

Seems it works well, thanks. If this is the way how to get complete program
trace including
daemons and programs that spawns child processes and if there isn''t
easier way how to get it
(without creating 3 scripts and with two destructive stop()''s i in the
way) than maybe it could be
added to DTraceTookit ?

Thanks, J.
> ------------ P?vodn? zpr?va ------------
> Od: Vladimir Marek <Vladimir.Marek at Sun.COM>
> P?edm?t: Re: [dtrace-discuss] Help pid provider follow child processes
> Datum: 07.5.2008 22:33:05
> ----------------------------------------
> > I just tried stop() as you suggested and I thing something bad is
happening
> > when you are using this action. Please see following. Having this two
very
> > simple dtrace scripts:
> > 
> > #!/usr/sbin/dtrace -wqs
> > /*this is master.d dscript file*/
> > 
> > proc:::exec-success
> 
> My unqualified guess is that exec-success probe is executed right after
> the process image is replaced by new one, which is way before any
> symbols are loaded from any libraries. So setting in the second script
> pid$1:::entry sets probes only in symbols defined in the binary itself.
> 
> I tried to create another scenario with third dtrace script in the
> middle between master and helper and it seems to work as requested.
> 
> I''m attaching the scripts.
> 
> On the side note, if you don''t want to see realtime linker in the
> output, you can try to run the binary with LD_BIND_NOW environment set
> (see man ld.so.1)
> 
> Hope this helps
> 
> -- 
> 	Vlad
> 
> 
>