dtrace discuss - Dec 2006 - Use pid provider probes for a varying number of processes

Hi all,

I''m curious if there is a means/workaround to use the pid provider 
probes for event tracking with a varying number of processes.

The case where I would need this is for event tracking with a bunch of 
concurrently launched processes as I get them when I run a script such as

    #!/bin/sh
    nsync=$1
    i=0

    while [ $i -lt $nsync ]; do
       qsub -sync y -b y /bin/sleep 1 &
       i=`expr $i + 1`
    done
    wait

that script launches me

  - 10 child processes when I pass ''10'' as argument
  - 100 child processes when I pass ''100'' as argument
  - 1000 child processes when I pass ''1000'' as argument

now what I would like to do is event tracking for any number 
of commands launched. I already tried with pid provider

    pid*::japi_init:entry

but this gets me

    probe description pid*::japi_init:entry does not match any probes

as error.

Any clue how to achieve this?

Thanks,
Andreas

It''s my understanding that the actual provider name for the pid
provider
(pidnnnn) is static, and is variable before D compilation proper, when 
macros are still being evaluated (i.e. pid$1).  So you could pass a long 
list of PIDs to your D program.

However, that''s not very flexible, as you''d have to pass the
same number
of PIDs each time, since the number of probe descriptions is fixed.

The next level would be to generate and invoke your D program from a 
shell script, so that you could generate probe descriptions on the fly.

Chip

Andreas.Haas at Sun.COM wrote:
> Hi all,
>
> I''m curious if there is a means/workaround to use the pid provider
> probes for event tracking with a varying number of processes.
>
> The case where I would need this is for event tracking with a bunch of 
> concurrently launched processes as I get them when I run a script such as
>
>    #!/bin/sh
>    nsync=$1
>    i=0
>
>    while [ $i -lt $nsync ]; do
>       qsub -sync y -b y /bin/sleep 1 &
>       i=`expr $i + 1`
>    done
>    wait
>
> that script launches me
>
>  - 10 child processes when I pass ''10'' as argument
>  - 100 child processes when I pass ''100'' as argument
>  - 1000 child processes when I pass ''1000'' as argument
>
> now what I would like to do is event tracking for any number of 
> commands launched. I already tried with pid provider
>
>    pid*::japi_init:entry
>
> but this gets me
>
>    probe description pid*::japi_init:entry does not match any probes
>
> as error.
>
> Any clue how to achieve this?
>
> Thanks,
> Andreas
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

On Fri, 1 Dec 2006, Chip Bennett wrote:
> The next level would be to generate and invoke your D program from a shell 
> script, so that you could generate probe descriptions on the fly.
Sure, that will always work

    (1) collect the pids of all processes launched with an
''&''
    (2) use the pid-list as input for generating one D script
        where all ''pid*''-probes are duplicated accordingly
    (3) run the generated D script using dtrace

I just thought there could be a more compfortable solution.

Thanks,
Andreas

On Fri, Dec 01, 2006 at 10:26:49AM -0600, Chip Bennett
wrote:
> It''s my understanding that the actual provider name for the pid
provider
> (pidnnnn) is static, and is variable before D compilation proper, when 
> macros are still being evaluated (i.e. pid$1).  So you could pass a long 
> list of PIDs to your D program.
That''s not quite right. The pid provider probes are created on the fly
by
libdtrace, but matching happens in the kernel as normal. By default there
are no pid provider probes:

# dtrace -n pid*:::
dtrace: invalid probe specifier pid*:::: probe description pid*::: does not
match any probes

But if I were to make some with other DTrace enablings:

# dtrace -n pid170011::main:entry
dtrace: description ''pid170011::main:entry'' matched 1 probe
^C
# dtrace -n pid170017::main:return
dtrace: description ''pid170017::main:return'' matched 1 probe
^C

The original probe description will match those probes:

# dtrace -n pid*:::
dtrace: description ''pid*:::'' matched 2 probes
^C
# dtrace -n pid*::: -l
   ID   PROVIDER            MODULE                          FUNCTION NAME
  455  pid170011              tcsh                              main entry
  456  pid170017              bash                              main return

We only create probes on demand when an explicit pid is provided; the
thinking is that actually sifting through every process isn''t ever what
you really want, and it wouldn''t pick up newly spawned processes.

One solution might be if we supported following traced processes on fork.
This is something we''ve wanted to do for some time, but it''s a
significant
project.

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

Well that''s pretty cool, and actually lends itself to a fairly elegant 
solution (IMHO).

Knowing that you are going to "probe" a process later, you could
create
the pid provider probes at the time you fork the process, with a dtrace 
command that just exits, something like:

    sleep 1000 &
    dtrace -q -n "BEGIN { exit(0) }" -n "pid$!:sleep:main:return
{}"

Later, another DTrace program can specify something like 
pid*:sleep:main:return and match all of the previously created 
background jobs.  You still have to pre-load all of the functions you 
want to trace, but it would make the final D program much simpler.

Chip


Adam Leventhal wrote:
> On Fri, Dec 01, 2006 at 10:26:49AM -0600, Chip Bennett wrote:
>   
>> It''s my understanding that the actual provider name for the
pid provider
>> (pidnnnn) is static, and is variable before D compilation proper, when 
>> macros are still being evaluated (i.e. pid$1).  So you could pass a
long
>> list of PIDs to your D program.
>>     
>
> That''s not quite right. The pid provider probes are created on the
fly by
> libdtrace, but matching happens in the kernel as normal. By default there
> are no pid provider probes:
>
> # dtrace -n pid*:::
> dtrace: invalid probe specifier pid*:::: probe description pid*::: does not
match any probes
>
> But if I were to make some with other DTrace enablings:
>
> # dtrace -n pid170011::main:entry
> dtrace: description ''pid170011::main:entry'' matched 1
probe
> ^C
> # dtrace -n pid170017::main:return
> dtrace: description ''pid170017::main:return'' matched 1
probe
> ^C
>
> The original probe description will match those probes:
>
> # dtrace -n pid*:::
> dtrace: description ''pid*:::'' matched 2 probes
> ^C
> # dtrace -n pid*::: -l
>    ID   PROVIDER            MODULE                          FUNCTION NAME
>   455  pid170011              tcsh                              main entry
>   456  pid170017              bash                              main return
>
> We only create probes on demand when an explicit pid is provided; the
> thinking is that actually sifting through every process isn''t ever
what
> you really want, and it wouldn''t pick up newly spawned processes.
>
> One solution might be if we supported following traced processes on fork.
> This is something we''ve wanted to do for some time, but
it''s a significant
> project.
>
> Adam
>
>

On Fri, Dec 01, 2006 at 12:55:42PM -0600, Chip Bennett
wrote:
> Well that''s pretty cool, and actually lends itself to a fairly
elegant
> solution (IMHO).
> 
> Knowing that you are going to "probe" a process later, you could
create
> the pid provider probes at the time you fork the process, with a dtrace 
> command that just exits, something like:
> 
>    sleep 1000 &
>    dtrace -q -n "BEGIN { exit(0) }" -n
"pid$!:sleep:main:return {}"
> 
> Later, another DTrace program can specify something like 
> pid*:sleep:main:return and match all of the previously created 
> background jobs.  You still have to pre-load all of the functions you 
> want to trace, but it would make the final D program much simpler.
You can run into problems there if the probes get cleaned up. This will
happen if the fasttrap module is unloaded or if you start bumping up against
the upper bound of fasttrap tracepoints (250,000 is the default, but it can
be increased in fasttrap.conf).

Adam

-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl