Dear Tracy, I''ve got sched:::on-cpu and sched:::off-cpu handled, but I''m not sure how to get what I want from sched:::wakeup. inside sched:::wakeup, "self" is the "waker" thread, but I want to print and update information about the "wakee" thread..., and all I have are its'' lwpsinfo and psinfo pointers, how do I get the wakee threads'' Thread-Local-Storage pointer ??? sched:::on-cpu { then = self->offstamp; now = timestamp; ... do various accounting with (now - then) ... self->onstamp = now; self->stype = SOBJ_NONE; } sched:::wakeup { lwpsinfo_t * wakee_lwpsinfo = (lwpsinfo_t*) args[0]; psinfo_t * wakee_psinfo = (psinfo_t*) args[1]; then = ???->offstamp; now = timestamp; ...do various accounting with (now - then) and (???->stype)... ???->offstamp = now; ???->stype = SOBJ_NONE; } sched:::off-cpu { then = self->onstamp; now = timestamp; ...do various accounting with (now - then) and (self->stype)... self->offstamp = now; self->stype=(curlwpsinfo->pr_state==SSLEEP ?curlwpsinfo->pr_stype: curlwpsinfo->pr_state == SONPROC? SOBJ_NONE : curlwpsinfo->pr_state == SRUN ? SOBJ_NONE : SOBJ_UNKNOWN); } sincerely, Sleepless.
Brendan Gregg - Sun Microsystems
2007-Mar-29 01:39 UTC
[dtrace-discuss] non-self pointers ???
G''Day Peter, On Wed, Mar 28, 2007 at 05:18:48PM -0700, Peter Lawrence wrote:> Dear Tracy, > I''ve got sched:::on-cpu and sched:::off-cpu handled, but > I''m not sure how to get what I want from sched:::wakeup. > > inside sched:::wakeup, "self" is the "waker" thread, but I want to > print and update information about the "wakee" thread..., > > and all I have are its'' lwpsinfo and psinfo pointers, how do I get > the wakee threads'' Thread-Local-Storage pointer ???sched::: event-to-event based accounting is demonstrated in Chapter 26 of the DTrace Guide on docs.sun.com. Passing information between events of different thread context is achieved with an associative array keyed on the thread address, args[0]->pr_addr, not by accessing other Thread-Local storage. Brendan -- Brendan [CA, USA]
Brendan, Ahha!, very cute. So, tell me, is "self->xyz" really just a short hand for xyz[curlwpsinfo->pr_addr] ? The documentation on self variables says something vague about this, but it would certainly help if it were precise. -Pete. ps, the following is a quote from the Doc (from just after the vague definition of self) which I''m guessing is more of a fall-out from the implementation rather than a useful requirement, especially if we''ld have to give this up to be precise about "self->xyz" being exactly equivalent to "xyz[curlwpsinfo->pr_addr]". "A thread''s identity is unique over the lifetime of the system: if the thread exits and the same operating system data structure is used to create a new thread, this thread does _not_ reuse the same DTrace thread-local storage identity." the reason I''m making this point is that if a precise definition of self were given, then I''ld consider that chapter to be the appropriate place to note how to workaround my original problem. Otherwise, to place the workaround in that chapter makes the language definition look a little bit conspicuously incomplete. YMMV, VWPBY, Hope I''m Not Offending Anyone Thats Not My Intention... and for reference, here''s the "vague" statement that preceeds it. "You can think of a thread-local variable as an associative array that is implicitly indexed by a tuple that describes the thread''s identity in the system." Gregg - Sun Microsystems wrote On 03/28/07 18:39,:> G''Day Peter, > > On Wed, Mar 28, 2007 at 05:18:48PM -0700, Peter Lawrence wrote: > >>Dear Tracy, >> I''ve got sched:::on-cpu and sched:::off-cpu handled, but >>I''m not sure how to get what I want from sched:::wakeup. >> >>inside sched:::wakeup, "self" is the "waker" thread, but I want to >>print and update information about the "wakee" thread..., >> >>and all I have are its'' lwpsinfo and psinfo pointers, how do I get >>the wakee threads'' Thread-Local-Storage pointer ??? > > > sched::: event-to-event based accounting is demonstrated in Chapter 26 > of the DTrace Guide on docs.sun.com. Passing information between events > of different thread context is achieved with an associative array keyed > on the thread address, args[0]->pr_addr, not by accessing other > Thread-Local storage. > > Brendan >
Brendan Gregg - Sun Microsystems
2007-Mar-29 04:21 UTC
[dtrace-discuss] non-self pointers ???
On Wed, Mar 28, 2007 at 07:25:41PM -0700, Peter Lawrence wrote:> Brendan, > Ahha!, very cute. So, tell me, is "self->xyz" really just a > short hand for xyz[curlwpsinfo->pr_addr] ? The documentation on > self variables says something vague about this, but it would certainly > help if it were precise.No. Associative arrays have a global scope, and Thread-Local variables have a thread scope. Quoting from Chapter 3, "Scalar Variables ... Associative Arrays ... Thread-Local Variables DTrace provides the ability to declare variable storage that is local to each operating system thread, as opposed to the global variables demonstrated earlier in this chapter." It is precise. That''s not to say things can''t be improved; it''s useful to hear if any of the text is causing confusion (even if the text is precise) - as what might be obvious to the experts that wrote the DTrace Guide may not be obvious to newbies. ... I''m sure the following will also interest you, /usr/include/sys/dtrace_impl.h - discusses variable scope. # dtrace -Sn ''proc:::exec { associative[curlwpsinfo->pr_addr]++; \ self->thread++; }'' DIFO 0x7e6080 returns D type (integer) (size 8) OFF OPCODE INSTRUCTION 00: 29010001 ldgs DT_VAR(256), %r1 ! DT_VAR(256) = "curthread" 01: 25000002 setx DT_INTEGER[0], %r2 ! 0x0 02: 04010201 sll %r1, %r2, %r1 03: 05010201 srl %r1, %r2, %r1 04: 0e010002 mov %r1, %r2 05: 33000000 flushts 06: 25000101 setx DT_INTEGER[1], %r1 ! 0x8 07: 31000102 pushtv DT_TYPE(0), %r1, %r2 ! DT_TYPE(0) = D type 08: 34050001 ldgaa DT_VAR(1280), %r1 ! DT_VAR(1280) = "associative" 09: 25000202 setx DT_INTEGER[2], %r2 ! 0x1 10: 07010202 add %r1, %r2, %r2 11: 36050002 stgaa %r2, DT_VAR(1280) ! DT_VAR(1280) = "associative" 12: 23000001 ret %r1 NAME ID KND SCP FLAG TYPE curthread 100 scl glb r D type (pointer) (size 8) associative 500 arr glb r/w D type (integer) (size 8) DIFO 0x787620 returns D type (integer) (size 8) OFF OPCODE INSTRUCTION 00: 2c050001 ldts DT_VAR(1280), %r1 ! DT_VAR(1280) = "thread" 01: 25000002 setx DT_INTEGER[0], %r2 ! 0x1 02: 07010202 add %r1, %r2, %r2 03: 2d050002 stts %r2, DT_VAR(1280) ! DT_VAR(1280) = "thread" 04: 23000001 ret %r1 NAME ID KND SCP FLAG TYPE thread 500 scl tls r/w D type (integer) (size 8) dtrace: description ''proc:::exec '' matched 1 probe ^C Brendan -- Brendan [CA, USA]
Brendan, thanks. let me see if I can ask my question in a better way. Solaris (hence dtrace) has a couple different ways of id''ing threads procfs.h lwpstatus_t.pr_lwpid lwpsinfo_t.pr_lwpid thread.h kthread_t.t_tid kthread_t.t_did pid==0 kthread_t.t_tid == 0, Always. kthread_t.t_did == "unique thread id for debuggers like mdb" pid!=0 kthread_t.t_tid == 1..N, (a 1-origin sequential numbering of threads within the process, which corresponds to prstat LWPID, and dtrace TID, and procfs lwpstats_t.pr_lwpid. kthread_t.t_did == "unique thread id for debuggers like mdb" So, the question is which, if any, of the above is used to implement dtrace''s "self", and if none of the above then what? One thing that is clear from experimenting with dtrace is that dtrace''s "tid" variable is the same as kthread_t.t_tid, in other words, for kernel-only threads (those that don''t have a LWP associated with them), the "tid" is always 0. thanks, -Pete Lawrence. Gregg - Sun Microsystems wrote On 03/28/07 21:21,:> On Wed, Mar 28, 2007 at 07:25:41PM -0700, Peter Lawrence wrote: > >>Brendan, >> Ahha!, very cute. So, tell me, is "self->xyz" really just a >>short hand for xyz[curlwpsinfo->pr_addr] ? The documentation on >>self variables says something vague about this, but it would certainly >>help if it were precise. > > > No. Associative arrays have a global scope, and Thread-Local variables > have a thread scope. > > Quoting from Chapter 3, > > "Scalar Variables > ... > > Associative Arrays > ... > > Thread-Local Variables > > DTrace provides the ability to declare variable storage that is local > to each operating system thread, as opposed to the global variables > demonstrated earlier in this chapter." > > It is precise. > > That''s not to say things can''t be improved; it''s useful to hear if any > of the text is causing confusion (even if the text is precise) - as what > might be obvious to the experts that wrote the DTrace Guide may not be > obvious to newbies. > > ... > > I''m sure the following will also interest you, > > /usr/include/sys/dtrace_impl.h - discusses variable scope. > > # dtrace -Sn ''proc:::exec { associative[curlwpsinfo->pr_addr]++; \ > self->thread++; }'' > > DIFO 0x7e6080 returns D type (integer) (size 8) > OFF OPCODE INSTRUCTION > 00: 29010001 ldgs DT_VAR(256), %r1 ! DT_VAR(256) = "curthread" > 01: 25000002 setx DT_INTEGER[0], %r2 ! 0x0 > 02: 04010201 sll %r1, %r2, %r1 > 03: 05010201 srl %r1, %r2, %r1 > 04: 0e010002 mov %r1, %r2 > 05: 33000000 flushts > 06: 25000101 setx DT_INTEGER[1], %r1 ! 0x8 > 07: 31000102 pushtv DT_TYPE(0), %r1, %r2 ! DT_TYPE(0) = D type > 08: 34050001 ldgaa DT_VAR(1280), %r1 ! DT_VAR(1280) = "associative" > 09: 25000202 setx DT_INTEGER[2], %r2 ! 0x1 > 10: 07010202 add %r1, %r2, %r2 > 11: 36050002 stgaa %r2, DT_VAR(1280) ! DT_VAR(1280) = "associative" > 12: 23000001 ret %r1 > > NAME ID KND SCP FLAG TYPE > curthread 100 scl glb r D type (pointer) (size 8) > associative 500 arr glb r/w D type (integer) (size 8) > > DIFO 0x787620 returns D type (integer) (size 8) > OFF OPCODE INSTRUCTION > 00: 2c050001 ldts DT_VAR(1280), %r1 ! DT_VAR(1280) = "thread" > 01: 25000002 setx DT_INTEGER[0], %r2 ! 0x1 > 02: 07010202 add %r1, %r2, %r2 > 03: 2d050002 stts %r2, DT_VAR(1280) ! DT_VAR(1280) = "thread" > 04: 23000001 ret %r1 > > NAME ID KND SCP FLAG TYPE > thread 500 scl tls r/w D type (integer) (size 8) > dtrace: description ''proc:::exec '' matched 1 probe > ^C > > Brendan >
Brendan Gregg - Sun Microsystems
2007-Mar-29 22:15 UTC
[dtrace-discuss] non-self pointers ???
G''Day Peter, On Thu, Mar 29, 2007 at 12:40:29PM -0700, Peter Lawrence wrote:> Brendan, > thanks. let me see if I can ask my question in a better way.You are asking questions relating to a sched::: based accounting script, which are answered by examples in the DTrace Guide, and now you are asking questions about thread ID internals. The questions seem similar, but for very different purposes. Are you trying to write a useful script, or are you trying to learn DTrace internals for the sake of DTrace internals? If you were trying to write a sched::: script, then the DTrace Guide is an excellent reference for this purpose. What is the script for? If I get a chance, I''ll write it for you and post it to the channel. I''m pretty good at DTrace, I learnt it from the DTrace Guide. :-) For learning DTrace internals, you should find the source itself serves as excellent documentation due to the numerous verbose descriptions throughout. For example, /usr/include/sys/dtrace.h /usr/include/sys/dtrace_impl.h Both have more lines of comments than source. Note the degree of documentation in the answers that follow...> Solaris (hence dtrace) has a couple different ways of id''ing threads > > procfs.h > lwpstatus_t.pr_lwpid > lwpsinfo_t.pr_lwpid > thread.h > kthread_t.t_tid > kthread_t.t_did > > pid==0 > kthread_t.t_tid == 0, Always. > kthread_t.t_did == "unique thread id for debuggers like mdb" > > pid!=0 > kthread_t.t_tid == 1..N, (a 1-origin sequential numbering of > threads within the process, which > corresponds to prstat LWPID, and > dtrace TID, and > procfs lwpstats_t.pr_lwpid. > kthread_t.t_did == "unique thread id for debuggers like mdb" > > > So, the question is which, if any, of the above is used to implement > dtrace''s "self", and if none of the above then what?Already documented in the source (read the block comment), /usr/src/uts/common/dtrace/dtrace.c, [...] /* * The key for a thread-local variable consists of the lower 61 bits of the * t_did, plus the 3 bits of the highest active interrupt above LOCK_LEVEL. * We add DIF_VARIABLE_MAX to t_did to assure that the thread key is never * equal to a variable identifier. This is necessary (but not sufficient) to * assure that global associative arrays never collide with thread-local * variables. To guarantee that they cannot collide, we must also define the * order for keying dynamic variables. That order is: * * [ key0 ] ... [ keyn ] [ variable-key ] [ tls-key ] * * Because the variable-key and the tls-key are in orthogonal spaces, there is * no way for a global variable key signature to match a thread-local key * signature. */ #define DTRACE_TLS_THRKEY(where) { \ uint_t intr = 0; \ uint_t actv = CPU->cpu_intr_actv >> (LOCK_LEVEL + 1); \ for (; actv; actv >>= 1) \ intr++; \ ASSERT(intr < (1 << 3)); \ (where) = ((curthread->t_did + DIF_VARIABLE_MAX) & \ (((uint64_t)1 << 61) - 1)) | ((uint64_t)intr << 61); \ } [...]> One thing that is clear from experimenting with dtrace is that > dtrace''s "tid" variable is the same as kthread_t.t_tid, in other > words, for kernel-only threads (those that don''t have a LWP associated > with them), the "tid" is always 0.Here is the definition of DIF_VAR_TID, /usr/src/uts/common/sys/dtrace.h, [...] /* * DTrace Intermediate Format (DIF) * * The following definitions describe the DTrace Intermediate Format (DIF), a * a RISC-like instruction set and program encoding used to represent * predicates and actions that can be bound to DTrace probes. The constants * below defining the number of available registers are suggested minimums; the * compiler should use DTRACEIOC_CONF to dynamically obtain the number of * registers provided by the current DTrace implementation. */ [...] #define DIF_VAR_TID 0x0117 /* (per-process) thread ID */ [...] And the association with "tid", /usr/src/lib/libdtrace/common/dt_open.c, [...] /* * Table of global identifiers. This is used to populate the global identifier * hash when a new dtrace client open occurs. For more info see dt_ident.h. * The global identifiers that represent functions use the dt_idops_func ops * and specify the private data pointer as a prototype string which is parsed * when the identifier is first encountered. These prototypes look like ANSI * C function prototypes except that the special symbol "@" can be used as a * wildcard to represent a single parameter of any type (i.e. any dt_node_t). * The standard "..." notation can also be used to represent varargs. An empty * parameter list is taken to mean void (that is, no arguments are permitted). * A parameter enclosed in square brackets (e.g. "[int]") denotes an optional * argument. */ static const dt_ident_t _dtrace_globals[] = { [...] { "tid", DT_IDENT_SCALAR, 0, DIF_VAR_TID, DT_ATTR_STABCMN, DT_VERS_1_0, &dt_idops_type, "id_t" }, [...] Finally the lookup of DIF_VAR_TID, /usr/src/uts/common/dtrace/dtrace.c, [...] /* * This function implements the DIF emulator''s variable lookups. The emulator * passes a reserved variable identifier and optional built-in array index. */ static uint64_t dtrace_dif_variable(dtrace_mstate_t *mstate, dtrace_state_t *state, uint64_t v, uint64_t ndx) { [...] case DIF_VAR_TID: /* * See comment in DIF_VAR_PID. */ if (DTRACE_ANCHORED(mstate->dtms_probe) && CPU_ON_INTR(CPU)) return (0); return ((uint64_t)curthread->t_tid); [...] There are other ways variables are made available to scripts; see the translators in /usr/lib/dtrace (eg, /usr/lib/dtrace/procfs.d). Brendan -- Brendan [CA, USA]
Brendan, wow! its really nice to be working with open sources like this. And I have to admit that I really admire the dtrace language, I haven''t seen many that are as elegant as it is. Thanks for taking the time to actually dig up the implementation of "tid", it both answers my question, and shows me where to look things up in the future. But I think I''ll take up your offer to help write some sched::: scripts anyway! -Pete. ps, I''ve spent an entire lifetime working on languages, compilers, and optimizations, so to me understanding how to use something like dtrace, and understanding how dtrace is designed/implemented are inseparable. Gregg - Sun Microsystems wrote On 03/29/07 15:15,:> G''Day Peter, > > On Thu, Mar 29, 2007 at 12:40:29PM -0700, Peter Lawrence wrote: > >>Brendan, >> thanks. let me see if I can ask my question in a better way. > > > You are asking questions relating to a sched::: based accounting script, > which are answered by examples in the DTrace Guide, and now you are asking > questions about thread ID internals. The questions seem similar, but for > very different purposes. Are you trying to write a useful script, or > are you trying to learn DTrace internals for the sake of DTrace internals? > > If you were trying to write a sched::: script, then the DTrace Guide is > an excellent reference for this purpose. What is the script for? If I > get a chance, I''ll write it for you and post it to the channel. I''m > pretty good at DTrace, I learnt it from the DTrace Guide. :-) > > For learning DTrace internals, you should find the source itself serves > as excellent documentation due to the numerous verbose descriptions > throughout. For example, > > /usr/include/sys/dtrace.h > /usr/include/sys/dtrace_impl.h > > Both have more lines of comments than source. Note the degree of > documentation in the answers that follow... > > >>Solaris (hence dtrace) has a couple different ways of id''ing threads >> >> procfs.h >> lwpstatus_t.pr_lwpid >> lwpsinfo_t.pr_lwpid >> thread.h >> kthread_t.t_tid >> kthread_t.t_did >> >>pid==0 >> kthread_t.t_tid == 0, Always. >> kthread_t.t_did == "unique thread id for debuggers like mdb" >> >>pid!=0 >> kthread_t.t_tid == 1..N, (a 1-origin sequential numbering of >> threads within the process, which >> corresponds to prstat LWPID, and >> dtrace TID, and >> procfs lwpstats_t.pr_lwpid. >> kthread_t.t_did == "unique thread id for debuggers like mdb" >> >> >>So, the question is which, if any, of the above is used to implement >>dtrace''s "self", and if none of the above then what? > > > Already documented in the source (read the block comment), > > /usr/src/uts/common/dtrace/dtrace.c, > [...] > /* > * The key for a thread-local variable consists of the lower 61 bits of the > * t_did, plus the 3 bits of the highest active interrupt above LOCK_LEVEL. > * We add DIF_VARIABLE_MAX to t_did to assure that the thread key is never > * equal to a variable identifier. This is necessary (but not sufficient) to > * assure that global associative arrays never collide with thread-local > * variables. To guarantee that they cannot collide, we must also define the > * order for keying dynamic variables. That order is: > * > * [ key0 ] ... [ keyn ] [ variable-key ] [ tls-key ] > * > * Because the variable-key and the tls-key are in orthogonal spaces, there is > * no way for a global variable key signature to match a thread-local key > * signature. > */ > #define DTRACE_TLS_THRKEY(where) { \ > uint_t intr = 0; \ > uint_t actv = CPU->cpu_intr_actv >> (LOCK_LEVEL + 1); \ > for (; actv; actv >>= 1) \ > intr++; \ > ASSERT(intr < (1 << 3)); \ > (where) = ((curthread->t_did + DIF_VARIABLE_MAX) & \ > (((uint64_t)1 << 61) - 1)) | ((uint64_t)intr << 61); \ > } > [...] > > >>One thing that is clear from experimenting with dtrace is that >>dtrace''s "tid" variable is the same as kthread_t.t_tid, in other >>words, for kernel-only threads (those that don''t have a LWP associated >>with them), the "tid" is always 0. > > > Here is the definition of DIF_VAR_TID, > > /usr/src/uts/common/sys/dtrace.h, > [...] > /* > * DTrace Intermediate Format (DIF) > * > * The following definitions describe the DTrace Intermediate Format (DIF), a > * a RISC-like instruction set and program encoding used to represent > * predicates and actions that can be bound to DTrace probes. The constants > * below defining the number of available registers are suggested minimums; the > * compiler should use DTRACEIOC_CONF to dynamically obtain the number of > * registers provided by the current DTrace implementation. > */ > [...] > #define DIF_VAR_TID 0x0117 /* (per-process) thread ID */ > [...] > > And the association with "tid", > > /usr/src/lib/libdtrace/common/dt_open.c, > [...] > /* > * Table of global identifiers. This is used to populate the global identifier > * hash when a new dtrace client open occurs. For more info see dt_ident.h. > * The global identifiers that represent functions use the dt_idops_func ops > * and specify the private data pointer as a prototype string which is parsed > * when the identifier is first encountered. These prototypes look like ANSI > * C function prototypes except that the special symbol "@" can be used as a > * wildcard to represent a single parameter of any type (i.e. any dt_node_t). > * The standard "..." notation can also be used to represent varargs. An empty > * parameter list is taken to mean void (that is, no arguments are permitted). > * A parameter enclosed in square brackets (e.g. "[int]") denotes an optional > * argument. > */ > static const dt_ident_t _dtrace_globals[] = { > [...] > { "tid", DT_IDENT_SCALAR, 0, DIF_VAR_TID, DT_ATTR_STABCMN, DT_VERS_1_0, > &dt_idops_type, "id_t" }, > [...] > > Finally the lookup of DIF_VAR_TID, > > /usr/src/uts/common/dtrace/dtrace.c, > [...] > /* > * This function implements the DIF emulator''s variable lookups. The emulator > * passes a reserved variable identifier and optional built-in array index. > */ > static uint64_t > dtrace_dif_variable(dtrace_mstate_t *mstate, dtrace_state_t *state, uint64_t v, > uint64_t ndx) > { > [...] > case DIF_VAR_TID: > /* > * See comment in DIF_VAR_PID. > */ > if (DTRACE_ANCHORED(mstate->dtms_probe) && CPU_ON_INTR(CPU)) > return (0); > > return ((uint64_t)curthread->t_tid); > [...] > > There are other ways variables are made available to scripts; see the > translators in /usr/lib/dtrace (eg, /usr/lib/dtrace/procfs.d). > > Brendan >