Hi,
we are inserting dtrace prones into our APL interpreter here (thanks
Angelo and Jignesh!) and things finally start working, but I am now seeing
a lot of messages of the type:
dtrace: error on enabled probe ID 7 (ID 42626: dsax21755:apl:xgoto0:af_return):
invalid address (0x10b) in predicate at DIF offset 28
dtrace: error on enabled probe ID 12 (ID 42626: dsax21755:apl:xgoto0:af_return):
invalid address (0x10b) in predicate at DIF offset 28
How can I find out what and where is producing these errors?
The script being used is (improvements welcomed, btw)
#!/usr/sbin/dtrace -s
BEGIN
{
start = timestamp;
}
dsax$1:::*_entry
{
fname = copyinstr(arg0);
@ce[fname] = count();
level[fname]++;
ts[fname] = timestamp;
}
dsax$1:::*_return
/level[copyinstr(arg0)] > 0 && ts[copyinstr(arg0)]/
{
fname = copyinstr(arg0);
@cr[fname] = count();
@t[fname] = sum(timestamp - ts[fname]);
level[fname]--;
ts[fname]=0;
}
dsax$1:::*_return
/level[copyinstr(arg0)] == 0 && ts[copyinstr(arg0)]/
{
fname = copyinstr(arg0);
@cr[fname] = count();
@t[fname] = sum(timestamp - ts[fname]);
ts[fname]=0;
}
END
{
normalize(@t, (timestamp - start)/1000000000);
printa(@ce);
printa(@cr);
printa(@t);
}
Any ideas what is producing the above errors? Is that related to the
script or to the position of dtrace probes in source code? How can I debug
this stuff?
Also, the above IDs, what do they correspond to? When I list all probes, I
get the following:
% dtrace -n dsax21755::: -l | tr -s '' ''
ID PROVIDER MODULE FUNCTION NAME
42623 dsax21755 apl xaunmd af_entry
42624 dsax21755 apl popsie af_return
42625 dsax21755 apl xgoton af_return
42626 dsax21755 apl xgoto0 af_return
42627 dsax21755 apl xaunmd if_entry
42628 dsax21755 apl xaunmd if_return
42629 dsax21755 apl xaunmd qdna_entry
42630 dsax21755 apl xaunmd qdna_return
42631 dsax21755 apl xaunmd ufncall
TIA, Dragan
--
Dragan Cvetkovic,
To be or not to be is true. G. Boole No it isn''t. L. E. J. Brouwer
Hi Dragan, Is it possible that you''re passing invalid pointers to your USDT probes? Try aggregating on arg0 for those dsax$1:::*_return probes and see if anything pops out. There are a couple of places where you probably want to be using probe-local or thread-local variables. In particular, fname should be probe-local (this->fname) and level may want to be thread-local (self->level) since you seem to be tracking the stack depth for a single thread. Adam On Fri, Sep 02, 2005 at 09:57:43AM -0400, Dragan Cvetkovic wrote:> Hi, > > we are inserting dtrace prones into our APL interpreter here (thanks > Angelo and Jignesh!) and things finally start working, but I am now seeing > a lot of messages of the type: > > dtrace: error on enabled probe ID 7 (ID 42626: > dsax21755:apl:xgoto0:af_return): invalid address (0x10b) in predicate at > DIF offset 28 > > dtrace: error on enabled probe ID 12 (ID 42626: > dsax21755:apl:xgoto0:af_return): invalid address (0x10b) in predicate at > DIF offset 28 > > How can I find out what and where is producing these errors? > > The script being used is (improvements welcomed, btw) > > #!/usr/sbin/dtrace -s > > BEGIN > { > start = timestamp; > } > > dsax$1:::*_entry > { > fname = copyinstr(arg0); > @ce[fname] = count(); > level[fname]++; > ts[fname] = timestamp; > } > > dsax$1:::*_return > /level[copyinstr(arg0)] > 0 && ts[copyinstr(arg0)]/ > { > fname = copyinstr(arg0); > @cr[fname] = count(); > @t[fname] = sum(timestamp - ts[fname]); > level[fname]--; > ts[fname]=0; > } > > dsax$1:::*_return > /level[copyinstr(arg0)] == 0 && ts[copyinstr(arg0)]/ > { > fname = copyinstr(arg0); > @cr[fname] = count(); > @t[fname] = sum(timestamp - ts[fname]); > ts[fname]=0; > } > > END > { > normalize(@t, (timestamp - start)/1000000000); > printa(@ce); > printa(@cr); > printa(@t); > } > > Any ideas what is producing the above errors? Is that related to the > script or to the position of dtrace probes in source code? How can I debug > this stuff? > > Also, the above IDs, what do they correspond to? When I list all probes, I > get the following: > > % dtrace -n dsax21755::: -l | tr -s '' '' > ID PROVIDER MODULE FUNCTION NAME > 42623 dsax21755 apl xaunmd af_entry > 42624 dsax21755 apl popsie af_return > 42625 dsax21755 apl xgoton af_return > 42626 dsax21755 apl xgoto0 af_return > 42627 dsax21755 apl xaunmd if_entry > 42628 dsax21755 apl xaunmd if_return > 42629 dsax21755 apl xaunmd qdna_entry > 42630 dsax21755 apl xaunmd qdna_return > 42631 dsax21755 apl xaunmd ufncall > > > TIA, Dragan > > -- > Dragan Cvetkovic, > > To be or not to be is true. G. Boole No it isn''t. L. E. J. Brouwer > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org-- Adam Leventhal, Solaris Kernel Development http://blogs.sun.com/ahl
On Fri, Sep 02, 2005 at 09:57:43AM -0400, Dragan Cvetkovic wrote:> Hi, > > we are inserting dtrace prones into our APL interpreter here (thanks > Angelo and Jignesh!)Wow -- very cool! I assume that this is Soliton''s SAX? Is it possible for people to kick the tires of the provider? I -- and presumably others -- would love to blog about this. In eager anticipation, I have downloaded SAX for Solaris x86 and am currently fooling around with it on my laptop. (It only took me about ten minutes to learn how to quit -- I think ")off" gives adb''s "$%" a run for its money as the strangest way to quit a program.) If you can drop me (or anyone) a binary with the provider, I''d love to play around with it...> and things finally start working, but I am now seeing > a lot of messages of the type: > > dtrace: error on enabled probe ID 7 (ID 42626: > dsax21755:apl:xgoto0:af_return): invalid address (0x10b) in predicate at > DIF offset 28 > > dtrace: error on enabled probe ID 12 (ID 42626: > dsax21755:apl:xgoto0:af_return): invalid address (0x10b) in predicate at > DIF offset 28As the message says, you have an invalid addres in the predicate for those return probes:> dsax$1:::*_return > /level[copyinstr(arg0)] > 0 && ts[copyinstr(arg0)]/ > { > fname = copyinstr(arg0); > @cr[fname] = count(); > @t[fname] = sum(timestamp - ts[fname]); > level[fname]--; > ts[fname]=0; > } > > dsax$1:::*_return > /level[copyinstr(arg0)] == 0 && ts[copyinstr(arg0)]/ > { > fname = copyinstr(arg0); > @cr[fname] = count(); > @t[fname] = sum(timestamp - ts[fname]); > ts[fname]=0; > }arg0 is bogus in some cases -- your provider isn''t correctly delivering a pointer to the function name in these cases. To understand why, I would first do this: # dtrace -n ''dsax*:::return{@[arg0] = count()}'' You''ll probably see some strange values in there (like, say, 0 or 267). You can then form another predicate with, say, a stop() action to figure out where exactly the bogus argument is coming from. Incidentally, I think you probably want to name your provider "sax" -- we view the "dvm" provider as somewhat anamolous in terms of naming; you should view the PHP and Ruby providers as a better model (they''re named "php" and "ruby", respectively). Anyway, hope this helps. This is very exciting stuff! - Bryan -------------------------------------------------------------------------- Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc
On Fri, 2 Sep 2005, Bryan Cantrill wrote:> > On Fri, Sep 02, 2005 at 09:57:43AM -0400, Dragan Cvetkovic wrote: >> Hi, >> >> we are inserting dtrace prones into our APL interpreter here (thanks >> Angelo and Jignesh!) > > Wow -- very cool! I assume that this is Soliton''s SAX? Is it possible > for people to kick the tires of the provider? I -- and presumably > others -- would love to blog about this. In eager anticipation, I have > downloaded SAX for Solaris x86 and am currently fooling around with it on > my laptop. (It only took me about ten minutes to learn how to quit -- I > think ")off" gives adb''s "$%" a run for its money as the strangest way to > quit a program.) If you can drop me (or anyone) a binary with the > provider, I''d love to play around with it...Hi Bryan, thanks for encouraging words, I will put a dtrace-aware version soon on our ftp site (beware, it''s compiled on x86 nv21!) and will let you know about that (correct, it''s Soliton''s SAX). Btw, some documentation is at ftp://ftp.tor.soliton.com/pub/SAXreleases/BetaRelease/documentation.tgz> >> and things finally start working, but I am now seeing >> a lot of messages of the type: >> >> dtrace: error on enabled probe ID 7 (ID 42626: >> dsax21755:apl:xgoto0:af_return): invalid address (0x10b) in predicate at >> DIF offset 28[snip]> > arg0 is bogus in some cases -- your provider isn''t correctly delivering > a pointer to the function name in these cases. To understand why, I > would first do this:OK, I *think* I have solved out this problem: not all functions that APL interpreter calls are named functions, so in some cases data I was passing to DTRACE_PROBE() was NULL (or similar). I am now guarding against such cases in my source code. [snip]> I think you probably want to name your provider "sax" -- we view the > "dvm" provider as somewhat anamolous in terms of naming; you should > view the PHP and Ruby providers as a better model (they''re named "php" > and "ruby", respectively).Will have a look, thanks. But for the time being, it''s still dsax.> Anyway, hope this helps. This is very exciting stuff!It does indeed. Thanks. Bye, Dragan P.S. I have just uploaded our dtrace-aware sax distribution, it''s on ftp://ftp.tor.soliton.com/pub/SAXreleases/BetaRelease/sax621-sunos511dtrace-x86.tar.gz Be careful, it is a bleeding edge version that needs to run on a bleeding edge O/S :-) -- Dragan Cvetkovic, To be or not to be is true. G. Boole No it isn''t. L. E. J. Brouwer
On Fri, 2 Sep 2005, Adam Leventhal wrote:> Hi Dragan, > > Is it possible that you''re passing invalid pointers to your USDT probes?Hi Adam, you are right. It turned out (as I mentioned in my answer to bmc) that not all APL functions executed are named functions (inserting printf(3c) before DTRACE_PROBE is a great way to figure that out since printf will dump core if a null pointer is passed -- then you just fire up dbx) so I am now checking for it before calling DTRACE_PROBEs.> There are a couple of places where you probably want to be using probe-local > or thread-local variables. In particular, fname should be probe-local > (this->fname) and level may want to be thread-local (self->level) since you > seem to be tracking the stack depth for a single thread.Well, we are trying to avoid issues with recursive functions which will hapily overwrite the previous dtrace data about them, so using level was a way of doing this. I''ll try your suggestion. Thanks and bye, Dragan -- Dragan Cvetkovic, To be or not to be is true. G. Boole No it isn''t. L. E. J. Brouwer
> >I think you probably want to name your provider "sax" -- we view the > >"dvm" provider as somewhat anamolous in terms of naming; you should > >view the PHP and Ruby providers as a better model (they''re named "php" > >and "ruby", respectively). > > Will have a look, thanks. But for the time being, it''s still dsax.Now that I have my mits on the provider: you''ll want to change the name from af_entry and if_entry to af-entry and if-entry. (You can do this with a double underscore in the name, i.e. it should be af__entry, if__entry, etc.) You''ll want to do this because I recently integrated a change to flowindent anything that ends with "-entry" or "-return". (Allowing Java, PHP and Ruby to be flowindented without hassle.) Here''s the bug ID, should you care: 6315975 flowindent should operate on any probe ending in "-entry"/"-return"> P.S. I have just uploaded our dtrace-aware sax distribution, it''s on > > ftp://ftp.tor.soliton.com/pub/SAXreleases/BetaRelease/sax621-sunos511dtrace-x86.tar.gz > > Be careful, it is a bleeding edge version that needs to run on a bleeding > edge O/S :-)I downloaded this, and the probes show up, but now I can''t seem to write the APL to tickle it. I was trying some simple functions like "roll" and "deal", and executed the statement on the Wikipedia page for APL (an 18 character (!) program that displays all of the primes between 2 and n), but I couldn''t seem to get any of the probes to fire. Is there some example APL I could run to watch the APL DTrace provider in action? - Bryan -------------------------------------------------------------------------- Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc
On Fri, 2 Sep 2005, Bryan Cantrill wrote:> >>> I think you probably want to name your provider "sax" -- we view the >>> "dvm" provider as somewhat anamolous in terms of naming; you should >>> view the PHP and Ruby providers as a better model (they''re named "php" >>> and "ruby", respectively). >> >> Will have a look, thanks. But for the time being, it''s still dsax. > > Now that I have my mits on the provider: you''ll want to change the name > from af_entry and if_entry to af-entry and if-entry. (You can do this > with a double underscore in the name, i.e. it should be af__entry, > if__entry, etc.) You''ll want to do this because I recently integrated > a change to flowindent anything that ends with "-entry" or "-return". > (Allowing Java, PHP and Ruby to be flowindented without hassle.) Here''s > the bug ID, should you care: > > 6315975 flowindent should operate on any probe ending in "-entry"/"-return"Thanks. Will do. Actually, I wanted to call them entry and return, but of course, the C compiler complains.>> P.S. I have just uploaded our dtrace-aware sax distribution, it''s on >> >> ftp://ftp.tor.soliton.com/pub/SAXreleases/BetaRelease/sax621-sunos511dtrace-x86.tar.gz >> >> Be careful, it is a bleeding edge version that needs to run on a bleeding >> edge O/S :-) > > I downloaded this, and the probes show up, but now I can''t seem to write > the APL to tickle it. I was trying some simple functions like "roll" and > "deal", and executed the statement on the Wikipedia page for APL (an 18 > character (!) program that displays all of the primes between 2 and n), > but I couldn''t seem to get any of the probes to fire. Is there some > example APL I could run to watch the APL DTrace provider in action?OK, you can download our, sort of, CPU benchmark program, suite: ftp://ftp.tor.soliton.com/pub/SAXreleases/BetaRelease/suite.sw and copy it e.g to your home directory. Once you download it, fire up APL interpreter: $ saxlite and from within it )load ~/suite driver 3 Press ^C once you had enough (it will stop after 256) and, as you have already discovered )off to exit the interpret. You''ll get a bunch of function calls. Or you can do e..g )load 1 unix (use )libs to see where APL think that 1 is mapped to) host ''uname -a'' to be on a more familiar ground :-) Dragan -- Dragan Cvetkovic, To be or not to be is true. G. Boole No it isn''t. L. E. J. Brouwer
> >>P.S. I have just uploaded our dtrace-aware sax distribution, it''s on > >> > >>ftp://ftp.tor.soliton.com/pub/SAXreleases/BetaRelease/sax621-sunos511dtrace-x86.tar.gz > >> > >>Be careful, it is a bleeding edge version that needs to run on a bleeding > >>edge O/S :-) > > > >I downloaded this, and the probes show up, but now I can''t seem to write > >the APL to tickle it. I was trying some simple functions like "roll" and > >"deal", and executed the statement on the Wikipedia page for APL (an 18 > >character (!) program that displays all of the primes between 2 and n), > >but I couldn''t seem to get any of the probes to fire. Is there some > >example APL I could run to watch the APL DTrace provider in action? > > OK, you can download our, sort of, CPU benchmark program, suite: > > ftp://ftp.tor.soliton.com/pub/SAXreleases/BetaRelease/suite.sw > > and copy it e.g to your home directory.Success -- very cool! (And is there a short answer as to why I couldn''t get it to work before with hand-rolled APL?) Obviously, I''m not an APL programmer (though I confess a kind of strange attraction -- it _is_ amazing how powerful the primitives are); do you anticipate the DTrace provider being able to shed light on APL programs? Very cool, at any rate... - Bryan -------------------------------------------------------------------------- Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc
On Fri, 2 Sep 2005, Bryan Cantrill wrote:>>> >>> I downloaded this, and the probes show up, but now I can''t seem to write >>> the APL to tickle it. I was trying some simple functions like "roll" and >>> "deal", and executed the statement on the Wikipedia page for APL (an 18 >>> character (!) program that displays all of the primes between 2 and n), >>> but I couldn''t seem to get any of the probes to fire. Is there some >>> example APL I could run to watch the APL DTrace provider in action? >> >> OK, you can download our, sort of, CPU benchmark program, suite: >> >> ftp://ftp.tor.soliton.com/pub/SAXreleases/BetaRelease/suite.sw >> >> and copy it e.g to your home directory. > > Success -- very cool! (And is there a short answer as to why I couldn''t > get it to work before with hand-rolled APL?)Well, one reason is that dtrace probes are only for user-defined functions. Primitives are not dtrace-d (yet). There is a Virtual Machine executing APL code and currently dtrace probes are only inserted in those instructions that are used to execute user-defined functions. That was the priority.> Obviously, I''m not an > APL programmer (though I confess a kind of strange attraction -- it _is_ > amazing how powerful the primitives are); do you anticipate the DTrace > provider being able to shed light on APL programs?I certainly hope so. Btw, I have tried changing af_entry and af_return to af-entry and af-return as in: if (cursie->fnstndx) { DTRACE_PROBE1(sax,af-return,fname); } but am getting the following compilation error messsages: ../xcode/xgoto0.c: In function ''xgoto0'': ../xcode/xgoto0.c:69: error: syntax error before ''-'' token ../xcode/xgoto0.c:69: error: ''__dtrace_sax___af'' undeclared (first use in this function) ../xcode/xgoto0.c:69: error: (Each undeclared identifier is reported only once ../xcode/xgoto0.c:69: error: for each function it appears in.) ../xcode/xgoto0.c:69: error: syntax error before ''return'' Is this supposed to be so :-) ?? Dragan -- Dragan Cvetkovic, To be or not to be is true. G. Boole No it isn''t. L. E. J. Brouwer
On Fri, Sep 02, 2005 at 04:31:14PM -0400, Dragan Cvetkovic wrote:> On Fri, 2 Sep 2005, Bryan Cantrill wrote: > >>> > >>>I downloaded this, and the probes show up, but now I can''t seem to write > >>>the APL to tickle it. I was trying some simple functions like "roll" and > >>>"deal", and executed the statement on the Wikipedia page for APL (an 18 > >>>character (!) program that displays all of the primes between 2 and n), > >>>but I couldn''t seem to get any of the probes to fire. Is there some > >>>example APL I could run to watch the APL DTrace provider in action? > >> > >>OK, you can download our, sort of, CPU benchmark program, suite: > >> > >>ftp://ftp.tor.soliton.com/pub/SAXreleases/BetaRelease/suite.sw > >> > >>and copy it e.g to your home directory. > > > >Success -- very cool! (And is there a short answer as to why I couldn''t > >get it to work before with hand-rolled APL?) > > Well, one reason is that dtrace probes are only for user-defined > functions. Primitives are not dtrace-d (yet). There is a Virtual Machine > executing APL code and currently dtrace probes are only inserted in those > instructions that are used to execute user-defined functions. That was the > priority. > > >Obviously, I''m not an > >APL programmer (though I confess a kind of strange attraction -- it _is_ > >amazing how powerful the primitives are); do you anticipate the DTrace > >provider being able to shed light on APL programs? > > I certainly hope so. > > Btw, I have tried changing af_entry and af_return to af-entry and > af-return as in: > > if (cursie->fnstndx) { > DTRACE_PROBE1(sax,af-return,fname);^ make this af__return> } >Cheers, - jonathan -- Jonathan Adams, Solaris Kernel Development
On Fri, 2 Sep 2005, Jonathan Adams wrote:> On Fri, Sep 02, 2005 at 04:31:14PM -0400, Dragan Cvetkovic wrote: >> >> Btw, I have tried changing af_entry and af_return to af-entry and >> af-return as in: >> >> if (cursie->fnstndx) { >> DTRACE_PROBE1(sax,af-return,fname); > ^ make this af__return >> } >>Thanks. I guess, the same holds for af-entry (i.e. af__entry). Bye, Dragan -- Dragan Cvetkovic, To be or not to be is true. G. Boole No it isn''t. L. E. J. Brouwer
On Fri, Sep 02, 2005 at 04:57:22PM -0400, Dragan Cvetkovic wrote:> On Fri, 2 Sep 2005, Jonathan Adams wrote: > > >On Fri, Sep 02, 2005 at 04:31:14PM -0400, Dragan Cvetkovic wrote: > >> > >>Btw, I have tried changing af_entry and af_return to af-entry and > >>af-return as in: > >> > >> if (cursie->fnstndx) { > >> DTRACE_PROBE1(sax,af-return,fname); > > ^ make this af__return > >> } > >> > > Thanks. I guess, the same holds for af-entry (i.e. af__entry).Yup; "__" is translated to "-" by dtrace. Cheers, - jonathan -- Jonathan Adams, Solaris Kernel Development
On Fri, 2 Sep 2005, Dragan Cvetkovic wrote:> > P.S. I have just uploaded our dtrace-aware sax distribution, it''s on > > ftp://ftp.tor.soliton.com/pub/SAXreleases/BetaRelease/sax621-sunos511dtrace-x86.tar.gz > > Be careful, it is a bleeding edge version that needs to run on a bleeding > edge O/S :-) >OK, I now have version both for SPARC (compiled on Solaris 10) and for i86pc (compiled on snv_22), incorporating all suggestions re provider name and using - (i.e. __) instead of _ in names. It also containes some new probes. Links are ftp://ftp.tor.soliton.com/pub/SAXreleases/BetaRelease/sax621-sunos510dtrace-sparc.tar.bz2 and ftp://ftp.tor.soliton.com/pub/SAXreleases/BetaRelease/sax621-sunos511dtrace-x86.tar.bz2 Bye, Dragan -- Dragan Cvetkovic, To be or not to be is true. G. Boole No it isn''t. L. E. J. Brouwer