dtrace discuss - Jul 2007 - I do not understand. Please advise.

I have the following script:

proc:::exec-success
/execname == "java"/
{
    stop();
    printf("Restart with prun %d\n", pid);
}

This one stops any java process that I start (and others :-). Runs as expected.
I "prun pid" and that stopped java process continues.

Now I add a single line directly after the printf:

    system("./scanit.d -p %d", pid);

scanit.d uses the hotspot probes to dive into this java process just started. 
If scanit.d is started from the command line it works as I expect it. However
when started through the above mentioned "system(...)" way, an error
is raised and the stopped java process is started again!

The message is:

dtrace: failed to compile script ./scanit.d: line 17: args[ ] may not be
referenced because probe description hotspot4051:::method-entry matches an
unstable set of probes

What am I missing? (and yes I did start the java process with
-XX:+ExtendedDTraceProbes)

Thanks
Paul
--
This message posted from opensolaris.org

Hi Paul,

It sounds like the ''scanit.d'' script is using the args[]
array, but should
instead be using argN variables, but it''s hard to say without seeing
the
script. Are you able to run ''scanit.d'' by hand?

Adam

On Wed, Jul 04, 2007 at 06:47:28AM -0700, Paul van den Bogaard
wrote:
> I have the following script:
> 
> proc:::exec-success
> /execname == "java"/
> {
>     stop();
>     printf("Restart with prun %d\n", pid);
> }
> 
> This one stops any java process that I start (and others :-). Runs as
expected. I "prun pid" and that stopped java process continues.
> 
> Now I add a single line directly after the printf:
> 
>     system("./scanit.d -p %d", pid);
> 
> scanit.d uses the hotspot probes to dive into this java process just
started.  If scanit.d is started from the command line it works as I expect it.
However when started through the above mentioned "system(...)" way, an
error is raised and the stopped java process is started again!
> 
> The message is:
> 
> dtrace: failed to compile script ./scanit.d: line 17: args[ ] may not be
referenced because probe description hotspot4051:::method-entry matches an
unstable set of probes
> 
> What am I missing? (and yes I did start the java process with
-XX:+ExtendedDTraceProbes)
> 
> Thanks
> Paul
> --
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Adam Leventhal, Solaris Kernel Development       http://blogs.sun.com/ahl

system(3) takes 1 argument only:

  int system(const char *command);

In your case, you will need to use sprintf(3) to generate the needed string.

Rayson



On 7/4/07, Paul van den Bogaard <Paul.VandenBogaard at sun.com>
wrote:
> I have the following script:
>
> proc:::exec-success
> /execname == "java"/
> {
>    stop();
>    printf("Restart with prun %d\n", pid);
> }
>
> This one stops any java process that I start (and others :-). Runs as
expected. I "prun pid" and that stopped java process continues.
>
> Now I add a single line directly after the printf:
>
>    system("./scanit.d -p %d", pid);
>
> scanit.d uses the hotspot probes to dive into this java process just
started.  If scanit.d is started from the command line it works as I expect it.
However when started through the above mentioned "system(...)" way, an
error is raised and the stopped java process is started again!
>
> The message is:
>
> dtrace: failed to compile script ./scanit.d: line 17: args[ ] may not be
referenced because probe description hotspot4051:::method-entry matches an
unstable set of probes
>
> What am I missing? (and yes I did start the java process with
-XX:+ExtendedDTraceProbes)
>
> Thanks
> Paul
> --
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>

Sorry, ignore my previous message, I was thinking in C/C++ :(

Rayson



On 7/4/07, Rayson Ho <rayrayson at gmail.com>
wrote:
> system(3) takes 1 argument only:
>
>  int system(const char *command);
>
> In your case, you will need to use sprintf(3) to generate the needed
string.
>
> Rayson
>
>
>
> On 7/4/07, Paul van den Bogaard <Paul.VandenBogaard at sun.com>
wrote:
> > I have the following script:
> >
> > proc:::exec-success
> > /execname == "java"/
> > {
> >    stop();
> >    printf("Restart with prun %d\n", pid);
> > }
> >
> > This one stops any java process that I start (and others :-). Runs as
expected. I "prun pid" and that stopped java process continues.
> >
> > Now I add a single line directly after the printf:
> >
> >    system("./scanit.d -p %d", pid);
> >
> > scanit.d uses the hotspot probes to dive into this java process just
started.  If scanit.d is started from the command line it works as I expect it.
However when started through the above mentioned "system(...)" way, an
error is raised and the stopped java process is started again!
> >
> > The message is:
> >
> > dtrace: failed to compile script ./scanit.d: line 17: args[ ] may not
be referenced because probe description hotspot4051:::method-entry matches an
unstable set of probes
> >
> > What am I missing? (and yes I did start the java process with
-XX:+ExtendedDTraceProbes)
> >
> > Thanks
> > Paul
> > --
> > This message posted from opensolaris.org
> > _______________________________________________
> > dtrace-discuss mailing list
> > dtrace-discuss at opensolaris.org
> >
>

*What

did some further testing on this subject. Please find here my findings and
ideas. Still open ended.

I am now stuck in my way of thinking. Like going around in circles.  I like to
instrument a process of which I know it will be created but I do not have the
pid yet. Looking for a creative idea that could help me get out of this chicken
and egg situation?

These tests are all performed on a single code AMD U20 workstation running
Solaris 10, Update 3.

*Why

I am trying to optimize a JDBC (pure Java) driver used for SPECjApps benchmarks.
The SPEC setup is quite complex. Therefore it would be very convenient if I
could make this stop in "proc:::exec" work in order to determine the
pid of the java process started and then start a next script  that needs this
pid.  This would enable me to focus on tracing specific classes (likely jdbc
related).

I have read about other methods to get a VM started with a pause option. Indeed
this works, however the stop() and go method I am trying to get working is
simpler in use.

*The DTrace scripts:

#!/usr/sbin/dtrace -Zs

#pragma D option quiet
#pragma D option destructive

int watchpid;

proc:::exec-success
/pid == watchpid/
{
        stop();
        printf("process \"%s\" %d stopped. Use prun %d to
activate\n",
                stringof(curpsinfo->pr_fname), pid,pid);
        system("./misf.d %d", pid);
}

proc:::exec-success
/execname == "java" && watchpid == 0/
{
        watchpid = pid;
}


*Some remarks

A java process re-execs itself so I like to get a hold of the second exec. I
think the above script takes care of that ''problem''. Now I
have a handle on the process. I stop the correct one.

Then it launches the script misf.d with as a single parameter the pid of that
java process.


*Interesting Observations:

1) if misf.d is started with the -p pid option  (and misf.d has probes specified
as "hotspot$target:::method-entry" than this automagically starts the
stopped process: no prun is needed.
2) when using it as in the code snippet above and handled like
"hotspot$1:::method-entry" I need to use prun to wake up the Java
process.

My adhoc impression: an undocumented feature.


*The misf.d script:

#!/usr/sbin/dtrace -Zs

#pragma D option quiet

self string class;

hotspot$1:::method-entry
{
        this->str_ptr = (char*) copyin(arg1, arg2+1);
        this->str_ptr[arg2] = ''\0'';
        self->class = (string) this->str_ptr;

        printf("%s\n", self->class);
}

Of course the original version is a little bit more complex, however this one
shows what is happening.


*What happens

If I remove the -Z option from the first line in misf.d (#!/usr/sbin/dtrace -Zs)
I get a DTrace error message telling me it found no matching probes. I think
this is due to the fact that all these hotspot probes are part of libjvm.so. A
library  that is not (completely?) loaded at this point in time. When the misf.d
script starts the java process is already stopped, therefore this DTrace script
does not find any probes.
And indeed leaving the -Z option in place results in absolutely no output at
all. Probes that do not exist can not be activated, so will never fire.


*Extra info on the hotspot provider

According to the docs there is a hotspot<pid>:::init-vm-end.  I was hoping
I could create a clause on the probe spec like
"hotspot*:::init-vm-end" and use stop() in this clause. However the
probe I am interested in is that of the JVM that still needs to be  started and
not of any other already started VM. Therefore my probe does not exist and
therefore can not be activated: I cannot catch my init-vm-end, since a
non-activated probe will never fire...


*Request for enhancement

Both for the hotspot and the pid provider: let the  probe specification in
DTrace scripts be like a template. Once a probe that matches the template is
created (after the DTrace script is started) it will be placed into instrumented
mode.
In my example this means I create an interst for the probe
"hotspot*:::vm-init-end". Start my DTrace script. And some time later
I start a JVM.  Since I expressed my interest automatically a probe for this
newly created process is activated so it will fire if this VM --in this
example-- finishes its initialisation.


*More tests

I also did a two stage stop. ./t.d to create the first stop. This time t.d did
not do the system() after that stop().
Then from a different window a

dtrace -wqn ''pidDDDDD::main:entry { stop(); }''

 (replace DDDD with the pid of the java proces I just learned)

Indeed this works but while this second stop is holding the app frozen a

dtrace -qn  ''hotspotDDDD:::method-entry { printf("Method
called\n"); }''

does indeed reproduce that error: no matching probes.

the above mentioned double stops need ofcourse a prun DDDD each to get any
progress ;^)

So even if I have a handle on the main() {} of the VM I still have no access to
the hotspot probes.


*The Java program

public class T {
        public int a() { return (int)(Math.random() * 1000.0);}
        public int b() { return a(); }
        public int c() { return b() + a(); }
        public int d(int a) { return a + c(); }

        static public void main(String[] args)  throws Exception {
                Thread.sleep(1000* Integer.parseInt(args[0]));
                System.out.println((new T()).d(1) );
        }
}

is started like

java -XX:+ExtendedDTraceProbes T 1

(you have to use Java SE 6 for this)


*Some remarks

when I replace the parameter with say 30 it sleeps 30 seconds before doing the
method calling. This gives me enough time to determine its pid and do a misf.d
from another shell. In this case the misf.d script behaves as I expect. This is
the reason why I think it could be related to library loading.
--
This message posted from opensolaris.org