dtrace discuss - Nov 2009 - dtracing a forked process OR dynamic library

Hi,

I am tracking down a problem and would like to know how I can follow a
forked process with my dtrace script, or how I can trace a dynamic library.

Here is the problem.  I am tracing dtlogin, and specifically I am trying
to determine what error libpkcs11`<routine> is returning.  It turns out
dtlogin forks a lot of processes, and I believe the second forked
process is the one that winds up having a failing call to
libpkcs11`<routine>.  Of course, when I start dtrace, the forked process
is not yet created, so I can not specify that.  Also, it appears I can
not just trace on libpkcs11`<routine> as libpkcs11 is only traceable
with a process that has it loaded.

So, the questions are:

How can I get dtrace to follow and trace forked processes?
How can I get dtrace to trace anything with libpkcs11 for any process
that is running or will be created and running in the future on a system?

Peter

Peter Shoults wrote:
> Hi,
>
> I am tracking down a problem and would like to know how I can follow a
> forked process with my dtrace script, or how I can trace a dynamic library.
>
> Here is the problem.  I am tracing dtlogin, and specifically I am trying
> to determine what error libpkcs11`<routine> is returning.  It turns
out
> dtlogin forks a lot of processes, and I believe the second forked
> process is the one that winds up having a failing call to
> libpkcs11`<routine>.  Of course, when I start dtrace, the forked
process
> is not yet created, so I can not specify that.  Also, it appears I can
> not just trace on libpkcs11`<routine> as libpkcs11 is only traceable
> with a process that has it loaded.
>
> So, the questions are:
>
> How can I get dtrace to follow and trace forked processes?
> How can I get dtrace to trace anything with libpkcs11 for any process
> that is running or will be created and running in the future on a system?
>   
Can''t you use the proc provider?

http://wikis.sun.com/display/DTrace/proc+Provider

There''s an example there that stashes the parent''s execname
(in your
case dtlogin) to find out what each parent calls and how often.
It''s not what you want but demonstrates tracing across the fork/exec 
boundary. I think you could adapt that for your purpose?

As for tracing libpkcs11 can you not construct a predicate based upon 
the dynamic library being loaded and that in turn enable the probes you 
want?

I''m probably very helpfully suggesting things here you''ve
tried -
apologies if that''s the case.
Cheers
PT.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3253 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20091119/e2527592/attachment.bin>

Check out this blog post from Chris Gerhard ...

http://blogs.sun.com/chrisg/entry/follow_fork_for_dtrace_pid

Peter Shoults wrote:
> Hi,
>
> I am tracking down a problem and would like to know how I can follow a
> forked process with my dtrace script, or how I can trace a dynamic library.
>
> Here is the problem.  I am tracing dtlogin, and specifically I am trying
> to determine what error libpkcs11`<routine> is returning.  It turns
out
> dtlogin forks a lot of processes, and I believe the second forked
> process is the one that winds up having a failing call to
> libpkcs11`<routine>.  Of course, when I start dtrace, the forked
process
> is not yet created, so I can not specify that.  Also, it appears I can
> not just trace on libpkcs11`<routine> as libpkcs11 is only traceable
> with a process that has it loaded.
>
> So, the questions are:
>
> How can I get dtrace to follow and trace forked processes?
> How can I get dtrace to trace anything with libpkcs11 for any process
> that is running or will be created and running in the future on a system?
>
> Peter
>
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>

Thank you Phil and Peter - I think this is just what I was looking for. 
I appreciate your help!

Peter

On 11/19/09 12:13, Phil Harman wrote:
> Check out this blog post from Chris Gerhard ...
>
> http://blogs.sun.com/chrisg/entry/follow_fork_for_dtrace_pid
>
> Peter Shoults wrote:
>> Hi,
>>
>> I am tracking down a problem and would like to know how I can follow a
>> forked process with my dtrace script, or how I can trace a dynamic
>> library.
>>
>> Here is the problem.  I am tracing dtlogin, and specifically I am
trying
>> to determine what error libpkcs11`<routine> is returning.  It
turns out
>> dtlogin forks a lot of processes, and I believe the second forked
>> process is the one that winds up having a failing call to
>> libpkcs11`<routine>.  Of course, when I start dtrace, the forked
process
>> is not yet created, so I can not specify that.  Also, it appears I can
>> not just trace on libpkcs11`<routine> as libpkcs11 is only
traceable
>> with a process that has it loaded.
>>
>> So, the questions are:
>>
>> How can I get dtrace to follow and trace forked processes?
>> How can I get dtrace to trace anything with libpkcs11 for any process
>> that is running or will be created and running in the future on a
>> system?
>>
>> Peter
>>
>>
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
>>   
>

why the need for a  stop() in that script ?

On Thu, Nov 19, 2009 at 12:13 PM, Phil Harman <phil.harman at gmail.com>
wrote:
> Check out this blog post from Chris Gerhard ...
>
> http://blogs.sun.com/chrisg/entry/follow_fork_for_dtrace_pid
>
> Peter Shoults wrote:
>>
>> Hi,
>>
>> I am tracking down a problem and would like to know how I can follow a
>> forked process with my dtrace script, or how I can trace a dynamic
>> library.
>>
>> Here is the problem. ?I am tracing dtlogin, and specifically I am
trying
>> to determine what error libpkcs11`<routine> is returning. ?It
turns out
>> dtlogin forks a lot of processes, and I believe the second forked
>> process is the one that winds up having a failing call to
>> libpkcs11`<routine>. ?Of course, when I start dtrace, the forked
process
>> is not yet created, so I can not specify that. ?Also, it appears I can
>> not just trace on libpkcs11`<routine> as libpkcs11 is only
traceable
>> with a process that has it loaded.
>>
>> So, the questions are:
>>
>> How can I get dtrace to follow and trace forked processes?
>> How can I get dtrace to trace anything with libpkcs11 for any process
>> that is running or will be created and running in the future on a
system?
>>
>> Peter
>>
>>
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
>>
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>

I suspect it is to stop the child process before it goes and does
whatever it is that we might want to capture.  I think the system() call
has something to do with getting it going again, but can''t find any
info
about system() yet.

pete

On 11/19/09 13:55, Max Levine wrote:
> why the need for a  stop() in that script ?
>
> On Thu, Nov 19, 2009 at 12:13 PM, Phil Harman <phil.harman at
gmail.com> wrote:
>   
>> Check out this blog post from Chris Gerhard ...
>>
>> http://blogs.sun.com/chrisg/entry/follow_fork_for_dtrace_pid
>>
>> Peter Shoults wrote:
>>     
>>> Hi,
>>>
>>> I am tracking down a problem and would like to know how I can
follow a
>>> forked process with my dtrace script, or how I can trace a dynamic
>>> library.
>>>
>>> Here is the problem.  I am tracing dtlogin, and specifically I am
trying
>>> to determine what error libpkcs11`<routine> is returning.  It
turns out
>>> dtlogin forks a lot of processes, and I believe the second forked
>>> process is the one that winds up having a failing call to
>>> libpkcs11`<routine>.  Of course, when I start dtrace, the
forked process
>>> is not yet created, so I can not specify that.  Also, it appears I
can
>>> not just trace on libpkcs11`<routine> as libpkcs11 is only
traceable
>>> with a process that has it loaded.
>>>
>>> So, the questions are:
>>>
>>> How can I get dtrace to follow and trace forked processes?
>>> How can I get dtrace to trace anything with libpkcs11 for any
process
>>> that is running or will be created and running in the future on a
system?
>>>
>>> Peter
>>>
>>>
>>> _______________________________________________
>>> dtrace-discuss mailing list
>>> dtrace-discuss at opensolaris.org
>>>
>>>       
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
>>
>>

Peter Shoults wrote:
> I suspect it is to stop the child process before it goes and does
> whatever it is that we might want to capture. 
right - not everything "in" dtrace happens synchronously (ie not all 
actions are executed at the time the probe fires - an example for that are 
the repeated reports of "I see hex numbers instead of symbols
reported"),
and stop makes sure the child process doesn''t go and do all the
interesting
work before dtrace gets instrumentation in place - or so I understand it.

Michael
-- 
Michael Schuster     http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''

A system() invocation is governed by buffer switchrate. The system() action
executes only when the containing buffer gets read. With default buffer policy
and switchrate, almost a second could go by before that happens.

So any process thread that fires the probe but terminates before the system()
action gets a chance will escape tracing. I tried grabbing ls(1) without using
stop(), for example, and got complained at every time.

There''s an example of deliberately stalling the system() action to foil
the date(1) command by upping the switchrate. P. 136 of the Dynamic Tracing
Guide (Sep 2008), part no 817-6223-12.
-- 
This message posted from opensolaris.org

On 11/19/09 15:00, michael schuster wrote:
> Peter Shoults wrote:
>> I suspect it is to stop the child process before it goes and does
>> whatever it is that we might want to capture. 
>
> right - not everything "in" dtrace happens synchronously (ie not
all
> actions are executed at the time the probe fires - an example for that
> are the repeated reports of "I see hex numbers instead of symbols
> reported"), and stop makes sure the child process doesn''t go
and do
> all the interesting work before dtrace gets instrumentation in place -
> or so I understand it.
>
> Michael
Ok - but how does one get that going again.....

Here is what I believe is my scenario....

PROC1 forks PROC2

PROC2 forks several other procs (I believe I am not interested in any of
these, but not 100% certain)

meanwhile...

PROC1 forks PROC3 (this I believe is the proc I am interested in)

What I want to be able to do is trace all the forked processes, but
especially PROC3.  I have found I can not use the self-> thread local
variable.  Wondering about setting an array to store ppid to predicate
on, but then I can''t think of how to get to predicate on an array of
unknown number of variables.

Pete

On Thu, Nov 19, 2009 at 02:02:53PM -0500, Peter Shoults
wrote:
> I suspect it is to stop the child process before it goes and does
> whatever it is that we might want to capture. 
That''s correct.
> I think the system() call
> has something to do with getting it going again, but can''t find
any info
> about system() yet.
system causes the dtrace userland process to fork and exec a command string.
Typically, you either do:

	stop();
	system("prun %d", pid);

or:

	stop();
	system("/path/to/dtrace/script -p %d", pid);

Both prun and dtrace will set the process running.

Cheers,
- jonathan
> pete
> 
> On 11/19/09 13:55, Max Levine wrote:
> > why the need for a  stop() in that script ?
> >
> > On Thu, Nov 19, 2009 at 12:13 PM, Phil Harman <phil.harman at
gmail.com> wrote:
> >   
> >> Check out this blog post from Chris Gerhard ...
> >>
> >> http://blogs.sun.com/chrisg/entry/follow_fork_for_dtrace_pid
> >>
> >> Peter Shoults wrote:
> >>     
> >>> Hi,
> >>>
> >>> I am tracking down a problem and would like to know how I can
follow a
> >>> forked process with my dtrace script, or how I can trace a
dynamic
> >>> library.
> >>>
> >>> Here is the problem.  I am tracing dtlogin, and specifically I
am trying
> >>> to determine what error libpkcs11`<routine> is
returning.  It turns out
> >>> dtlogin forks a lot of processes, and I believe the second
forked
> >>> process is the one that winds up having a failing call to
> >>> libpkcs11`<routine>.  Of course, when I start dtrace,
the forked process
> >>> is not yet created, so I can not specify that.  Also, it
appears I can
> >>> not just trace on libpkcs11`<routine> as libpkcs11 is
only traceable
> >>> with a process that has it loaded.
> >>>
> >>> So, the questions are:
> >>>
> >>> How can I get dtrace to follow and trace forked processes?
> >>> How can I get dtrace to trace anything with libpkcs11 for any
process
> >>> that is running or will be created and running in the future
on a system?
> >>>
> >>> Peter
> >>>
> >>>
> >>> _______________________________________________
> >>> dtrace-discuss mailing list
> >>> dtrace-discuss at opensolaris.org
> >>>
> >>>       
> >> _______________________________________________
> >> dtrace-discuss mailing list
> >> dtrace-discuss at opensolaris.org
> >>
> >>     
> 
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

Hey Pete:

If you need to look at processes that are in the process tree of a given
process. (ie) child of a child process etc, then you can use the progenyof()
action.

You would replace the predicate /ppid == $target/ by /progenyof($target)/

HTHs

Angelo


On Nov 19, 2009, at 3:15 PM, Peter Shoults wrote:
> On 11/19/09 15:00, michael schuster wrote:
>> Peter Shoults wrote:
>>> I suspect it is to stop the child process before it goes and does
>>> whatever it is that we might want to capture. 
>> 
>> right - not everything "in" dtrace happens synchronously (ie
not all
>> actions are executed at the time the probe fires - an example for that
>> are the repeated reports of "I see hex numbers instead of symbols
>> reported"), and stop makes sure the child process doesn''t
go and do
>> all the interesting work before dtrace gets instrumentation in place -
>> or so I understand it.
>> 
>> Michael
> Ok - but how does one get that going again.....
> 
> Here is what I believe is my scenario....
> 
> PROC1 forks PROC2
> 
> PROC2 forks several other procs (I believe I am not interested in any of
> these, but not 100% certain)
> 
> meanwhile...
> 
> PROC1 forks PROC3 (this I believe is the proc I am interested in)
> 
> What I want to be able to do is trace all the forked processes, but
> especially PROC3.  I have found I can not use the self-> thread local
> variable.  Wondering about setting an array to store ppid to predicate
> on, but then I can''t think of how to get to predicate on an array
of
> unknown number of variables.
> 
> Pete
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

Angelo Rajadurai wrote:
> Hey Pete:
>
> If you need to look at processes that are in the process tree of a given
process. (ie) child of a child process etc, then you can use the progenyof()
action.
>
> You would replace the predicate /ppid == $target/ by /progenyof($target)/
>
> HTHs
>
> Angelo
>
>
>   
An example of which might be :

$ more rac1.d
#pragma D option destructive

syscall::*fork*:return
/progenyof($target) && pid!=$target/
{
    stop();
    printf("%s, %d, %d", execname, pid, curpsinfo->pr_ppid);
    system("/rayh/t_open.d %d &", pid);
    exit(0);
}

and t_open.d looks like:

$ more t_open.d
#!/usr/sbin/dtrace -Fs

#pragma D option destructive
#pragma D option flowindent

BEGIN
{
        printf("prun''ing process %d\n", $1);
        system("prun %d\n", $1);
}

pid$1::mdmn_resume:entry
{
    self->follow = 1;
}

pid$1::_tx_open:entry
/self->follow/
{
    self->path = arg0;
    printf("%s\n", stringof(copyinstr(self->path)));
}

pid$1::_tx_open:return
/self->follow && (int)arg0 == -1/
{
    printf("errno : %d\n", errno);
    ustack();
}

pid$1::mdmn_resume:return
{
    self->follow = 0;
}

I would also advise taking a look at the Sun Developer pages on sunsolve 
or bigadmin if you have such access.

Hope this is what you are looking for
ray
> On Nov 19, 2009, at 3:15 PM, Peter Shoults wrote:
>
>   
>> On 11/19/09 15:00, michael schuster wrote:
>>     
>>> Peter Shoults wrote:
>>>       
>>>> I suspect it is to stop the child process before it goes and
does
>>>> whatever it is that we might want to capture. 
>>>>         
>>> right - not everything "in" dtrace happens synchronously
(ie not all
>>> actions are executed at the time the probe fires - an example for
that
>>> are the repeated reports of "I see hex numbers instead of
symbols
>>> reported"), and stop makes sure the child process
doesn''t go and do
>>> all the interesting work before dtrace gets instrumentation in
place -
>>> or so I understand it.
>>>
>>> Michael
>>>       
>> Ok - but how does one get that going again.....
>>
>> Here is what I believe is my scenario....
>>
>> PROC1 forks PROC2
>>
>> PROC2 forks several other procs (I believe I am not interested in any
of
>> these, but not 100% certain)
>>
>> meanwhile...
>>
>> PROC1 forks PROC3 (this I believe is the proc I am interested in)
>>
>> What I want to be able to do is trace all the forked processes, but
>> especially PROC3.  I have found I can not use the self-> thread
local
>> variable.  Wondering about setting an array to store ppid to predicate
>> on, but then I can''t think of how to get to predicate on an
array of
>> unknown number of variables.
>>
>> Pete
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
>>     
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>