dtrace discuss - May 2008 - Dtrace queries - predicates & func arg tracing

[1] Predicates in one-liners
I would like to list the probe modules in my executable and then
dynamically create a dscript to trace execution of those modules alone
(by excluding the 3rd party and system libraries). I tried the below
script without success. The conditional given in the predicate is not
taking effect. Why is this so ?
$ dtrace -ln ''pid$target::: /probemod!="libc.so.1"/ {
printf("%s -----
%s",probefunc, probename); } '' -p `pgrep a.out`

Similar is the case for the below script, I expect only the entry
points to be printed but all the probes in a.out are being printed.
$ dtrace -ln ''pid$target:a.out:: /probename=="entry"/ {
printf("%s
----- %s",probefunc, probename); } '' -p `pgrep a.out`

[2] Is there any means to get the arguments of a function dynamically?
I would like to create dscripts automatically to trace the functions
and if possible trace arguments as well (when I have access only to
the binaries).

TIA

regards
Shiv

Hi,
> [1] Predicates in one-liners
> I would like to list the probe modules in my executable and then
> dynamically create a dscript to trace execution of those modules alone
> (by excluding the 3rd party and system libraries). I tried the below
> script without success. The conditional given in the predicate is not
> taking effect. Why is this so ?
> $ dtrace -ln ''pid$target::: /probemod!="libc.so.1"/ {
printf("%s -----
> %s",probefunc, probename); } '' -p `pgrep a.out`
By using pid$target::: you already placed probe to every instruction in
the binary. And dtrace -l shows all of them. The condition /.../ is
evaluated when the probe is fired.

> Similar is the case for the below script, I expect only the entry
> points to be printed but all the probes in a.out are being printed.
> $ dtrace -ln ''pid$target:a.out:: /probename=="entry"/ {
printf("%s
> ----- %s",probefunc, probename); } '' -p `pgrep a.out`
I think that you want something like this:

dtrace -l -n ''pid$target:xxd::entry'' -c xxd
   ID   PROVIDER            MODULE                          FUNCTION NAME
76158   pid11781               xxd                            _start entry
76159   pid11781               xxd                             __fsr entry
76160   pid11781               xxd                   exit_with_usage entry
76161   pid11781               xxd                           huntype entry
76162   pid11781               xxd                           xxdline entry
76163   pid11781               xxd                              main entry


> [2] Is there any means to get the arguments of a function dynamically?
> I would like to create dscripts automatically to trace the functions
> and if possible trace arguments as well (when I have access only to
> the binaries).
$ dtrace -n ''pid$target::strcmp:entry{trace(copyinstr(arg0));
trace(copyinstr(arg1))}'' -c ls | tail
dtrace: description ''pid$target::strcmp:entry'' matched 2
probes
dtrace: pid 11814 has exited
dtrace: error on enabled probe ID 1 (ID 76165: pid11814:libc.so.1:strcmp:entry):
invalid address (0xfef1ca59) in action #2 at DIF offset 28
  1  76164                     strcmp:entry   pthread_getspecific               
pthread_getspecific
  1  76164                     strcmp:entry   flush                             
flush
  1  76164                     strcmp:entry   flush                             
flush
  1  76164                     strcmp:entry   _environ_lock                     
xflsbuf
  1  76164                     strcmp:entry   edata                             
xflsbuf
  1  76164                     strcmp:entry   xflsbuf                           
xflsbuf
  1  76164                     strcmp:entry   PROCEDURE_LINKAGE_TABLE_          
write
  1  76164                     strcmp:entry   write                             
write
  1  76164                     strcmp:entry   write                             
write

Let''s look at the script

pid$target::strcmp:entry - trace entry point of strcmp function (function
arguments are available in entry point)

copyinstr(arg0); - dtrace probes "are executed in kernel". You have to
  copy the string from userland to kernel by copyinstr function

trace(...) - dump something on the screen

So basically I just dump first and second parameter of the strcmp
function.

You can also trace return values:

$ dtrace -n ''pid$target::strcmp:entry{trace(copyinstr(arg0));
trace(copyinstr(arg1))} pid$target::strcmp:return{trace(arg1)}'' -c ls |
tail
dtrace: description ''pid$target::strcmp:entry'' matched 4
probes
dtrace: pid 11827 has exited
dtrace: error on enabled probe ID 1 (ID 76165: pid11827:libc.so.1:strcmp:entry):
invalid address (0xfef1ca59) in action #2 at DIF offset 28
  1  76166                    strcmp:return        4294967277
  1  76164                     strcmp:entry   xflsbuf                           
xflsbuf
  1  76166                    strcmp:return                 0
  1  76164                     strcmp:entry   PROCEDURE_LINKAGE_TABLE_          
write
  1  76166                    strcmp:return        4294967257
  1  76164                     strcmp:entry   write                             
write
  1  76166                    strcmp:return                 0
  1  76164                     strcmp:entry   write                             
write
  1  76166                    strcmp:return                 0


trace(arg1) displays the return value (only available in the *:return
probe)

Hope this helps

-- 
	Vlad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 193 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080520/deb6b5bb/attachment.bin>

Thanks a lot for the detailed response.

On Tue, May 20, 2008 at 4:13 PM, Vladimir Marek <Vladimir.Marek at
sun.com> wrote:
> Hi,
>
>> [1] Predicates in one-liners
>> I would like to list the probe modules in my executable and then
>> dynamically create a dscript to trace execution of those modules alone
>> (by excluding the 3rd party and system libraries). I tried the below
>> script without success. The conditional given in the predicate is not
>> taking effect. Why is this so ?
>> $ dtrace -ln ''pid$target::: /probemod!="libc.so.1"/
{ printf("%s -----
>> %s",probefunc, probename); } '' -p `pgrep a.out`
>
> By using pid$target::: you already placed probe to every instruction in
> the binary. And dtrace -l shows all of them. The condition /.../ is
> evaluated when the probe is fired.
>
This is a useful piece of information. Does placing the probe itself
have *any* overheads. For ex:
If I have 5 probefunc funcv, funcw, funcx, funcy & funcz and I am
interested in dtracing first 3, which in the below is a better,
-----------------------------------
Option 1:
pid$target:::entry
/(probefunc != funcy) && (probefunc != funcz)/
{}
-----------------------------------
Option 2:
pid$target::funcv:entry
{}

pid$target::funcw:entry
{}

pid$target::funcx:entry
{}
-----------------------------------

This is considering that the actual applications may have a few
hundred probemods and a few thousand probefuncs and I might be
interested in a subset of the available ones.
>
>> [2] Is there any means to get the arguments of a function dynamically?
>> I would like to create dscripts automatically to trace the functions
>> and if possible trace arguments as well (when I have access only to
>> the binaries).
>
> $ dtrace -n ''pid$target::strcmp:entry{trace(copyinstr(arg0));
trace(copyinstr(arg1))}'' -c ls | tail
>
<snip>
> pid$target::strcmp:entry - trace entry point of strcmp function (function
arguments are available in entry point)
>
<snip>
> So basically I just dump first and second parameter of the strcmp
> function.
>
<snip>
>
The only difficulty I see is that I need to know the number of
arguments and the type beforehand.

I have a simple program as below:
----------------------------------------------------
#include <stdio.h>

int foo(int n)
{  bar(n*n); return 0; }

int bar(int n) { return 0; }

int main()
{
	while (1)
	{
		foo(10); bar(10); sleep(2);
	}
	return 0;
}
----------------------------------------------------
I tried the below actions (in the return probe) without success:

printf("arg = %d",(int) copyin(arg0,sizeof(int)));
printf("arg = %d",(int) copyin(arg1,sizeof(int)));

trace(copyinstr(arg0));
trace(copyinstr(arg1));
trace(copyin(arg0,4));
trace(copyin(arg1,4));

I get an error equivalent to below in each case:
"dtrace: error on enabled probe ID 6574 (ID 87739:
pid795:a.out:bar:return): invalid address (0x1e) in action #7 at DIF
offset 40"

The DTrace guide suggests using copyinstr and copyin in the return
section instead of entry to ensure it is a faulted-in page, this I am
doing.

-Shiv

S h i v wrote:
> Thanks a lot for the detailed response.
> 
> On Tue, May 20, 2008 at 4:13 PM, Vladimir Marek <Vladimir.Marek at
sun.com> wrote:
>> Hi,
>>
>>> [1] Predicates in one-liners
>>> I would like to list the probe modules in my executable and then
>>> dynamically create a dscript to trace execution of those modules
alone
>>> (by excluding the 3rd party and system libraries). I tried the
below
>>> script without success. The conditional given in the predicate is
not
>>> taking effect. Why is this so ?
>>> $ dtrace -ln ''pid$target:::
/probemod!="libc.so.1"/ { printf("%s -----
>>> %s",probefunc, probename); } '' -p `pgrep a.out`
>> By using pid$target::: you already placed probe to every instruction in
>> the binary. And dtrace -l shows all of them. The condition /.../ is
>> evaluated when the probe is fired.
>>
> 
> This is a useful piece of information. Does placing the probe itself
> have *any* overheads. For ex:
yes, there is - it''s minimal, but it adds up. your option 1 means
"enable
all entry points in the target process" (ie actually fire(!)) "and
don''t do
anyting if probefunc is funcy or funcz", whereas 2 fires only at three 
entry points.
The difference is probably noticable.

you might consider 1a,

pid$target::func*:entry
/(probefunc != funcy) && (probefunc != funcz)/
{}

which, assuming you have only 5 functions matching the pattern
"func*",
will cause 5 probes to fire and not do anything for 2 of those (and perform 
the default action otherwise).
> If I have 5 probefunc funcv, funcw, funcx, funcy & funcz and I am
> interested in dtracing first 3, which in the below is a better,
> -----------------------------------
> Option 1:
> pid$target:::entry
> /(probefunc != funcy) && (probefunc != funcz)/
> {}
> -----------------------------------
> Option 2:
> pid$target::funcv:entry
> {}
> 
> pid$target::funcw:entry
> {}
> 
> pid$target::funcx:entry
> {}
> -----------------------------------
> 
> This is considering that the actual applications may have a few
> hundred probemods and a few thousand probefuncs and I might be
> interested in a subset of the available ones.
> 
>>> [2] Is there any means to get the arguments of a function
dynamically?
>>> I would like to create dscripts automatically to trace the
functions
>>> and if possible trace arguments as well (when I have access only to
>>> the binaries).
>> $ dtrace -n ''pid$target::strcmp:entry{trace(copyinstr(arg0));
trace(copyinstr(arg1))}'' -c ls | tail
>>
> <snip>
>> pid$target::strcmp:entry - trace entry point of strcmp function
(function arguments are available in entry point)
>>
> <snip>
>> So basically I just dump first and second parameter of the strcmp
>> function.
>>
> <snip>
> 
> The only difficulty I see is that I need to know the number of
> arguments and the type beforehand.
> 
> I have a simple program as below:
> ----------------------------------------------------
> #include <stdio.h>
> 
> int foo(int n)
> {  bar(n*n); return 0; }
> 
> int bar(int n) { return 0; }
> 
> int main()
> {
> 	while (1)
> 	{
> 		foo(10); bar(10); sleep(2);
> 	}
> 	return 0;
> }
> ----------------------------------------------------
> I tried the below actions (in the return probe) without success:
> 
> printf("arg = %d",(int) copyin(arg0,sizeof(int)));
> printf("arg = %d",(int) copyin(arg1,sizeof(int)));
> 
> trace(copyinstr(arg0));
> trace(copyinstr(arg1));
> trace(copyin(arg0,4));
> trace(copyin(arg1,4));
you don''t have to copyin() integers - integers are passed by value. 
copyin() and friends is necessary for data that''s passed by reference
(ie
pointer) across the userland/kernel boundary.

what you''re trying to do above basically amounts to dereferencing an 
integer as a pointer to integer.

HTH
Michael
-- 
Michael Schuster     http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''

Shiv,

S h i v wrote:
> [2] Is there any means to get the arguments of a function dynamically?
> I would like to create dscripts automatically to trace the functions
> and if possible trace arguments as well (when I have access only to
> the binaries).
>   
This is a very interesting area :)
I was looking at this option for a network tracing utility. But, 
realized that this is a generic problem.

"dtrace -lv " gives out the details of the args for every probes. Now 
this can be narrowed down to what
you want and it should be technically possible to generate the dtrace 
script dynamically.

Cheers,
Sanjeev.
> TIA
>
> regards
> Shiv
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>   
-- 
Solaris Revenue Products Engineering,
India Engineering Center,
Sun Microsystems India Pvt Ltd.
Tel:    x27521 +91 80 669 27521

On Thu, May 22, 2008 at 10:31 AM, Sanjeev Bagewadi
<Sanjeev.Bagewadi at sun.com> wrote:
> Shiv,
>
> This is a very interesting area :)
> I was looking at this option for a network tracing utility. But, realized
> that this is a generic problem.
>
> "dtrace -lv " gives out the details of the args for every probes.
Now this
> can be narrowed down to what
> you want and it should be technically possible to generate the dtrace
script
> dynamically.
>
Are there any conditions under which it provides the details. In my
case the entries where as below(provided for one of the functions):
------------------------------------------------------------------------------------
84452     pid763             a.out                               foo return

        Probe Description Attributes
                Identifier Names: Private
                Data Semantics:   Private
                Dependency Class: Unknown

        Argument Attributes
                Identifier Names: Private
                Data Semantics:   Private
                Dependency Class: Unknown

        Argument Types
                None
------------------------------------------------------------------------------------

I ran this test with the functions in a different libraries(assuming
the .so file might export some information) - a.out linking to
libexternal.so where the .so file had the functions implemented.
Result was no different.
I will keep a lookout for alternatives.

thanks
Shiv

On Thu, May 22, 2008 at 08:23:27PM +0530, S h i v wrote:
> Are there any conditions under which it provides the details. In my
> case the entries where as below(provided for one of the functions):
>
------------------------------------------------------------------------------------
> 84452     pid763             a.out                               foo return
> 
>         Probe Description Attributes
>                 Identifier Names: Private
>                 Data Semantics:   Private
>                 Dependency Class: Unknown
> 
>         Argument Attributes
>                 Identifier Names: Private
>                 Data Semantics:   Private
>                 Dependency Class: Unknown
> 
>         Argument Types
>                 None
>
------------------------------------------------------------------------------------
> 
> I ran this test with the functions in a different libraries(assuming
> the .so file might export some information) - a.out linking to
> libexternal.so where the .so file had the functions implemented.
> Result was no different.
> I will keep a lookout for alternatives.
Arguments for fbt probes, for example, are derived from CTF data that''s
compiled into the kernel. While Solaris libraries are compiled with CTF data,
DTrace doesn''t yet include that information in its probes. For the
moment,
you''ll have to determine the type of arguments by hand and use the argN
variables casting them as appropriate and bearing in mind potential bitness
differences between applications and the kernel.

I hope that helps.

Adam

-- 
Adam Leventhal, Fishworks                     http://blogs.sun.com/ahl