dtrace discuss - Jan 2008 - Help on refining pid probe in a multi-threaded process

Hello Dtrace community,
Firstly dtrace rocks!  It''s fun to play `from a bird''s eye`  
perspective generating different observations:  performance,  
debugging, learning, and outright curiosity etc

Here''s my problem using a 2 cpu AMD64 opteron desktop:
I''m experimenting with non-determinism using threads that get created  
dynamically depending on runtime contexts. The threading works fine  
and runs properly. Where i''m having debugging bouts is when
i''m
optimizing out threads for procedure calls. If the context has only 1  
thread to call, why launch it as a thread? Threading in my non- 
deterministic context is approximately 5 times slower then procedure  
calls. So this my experiment in optimizing out threads --- out damn  
spot.

Here''s a simple scenario where all the code is in C++:
			process p3
					|
	----------------------------------------
	|                                                   |
	V                                                 V     p3  
dynamically launches threads 1 and 2 at the same time: they are now  
parallelizing and report back to p3
        Th1                                            Th2
                                                            ^ then  
Th2 in it''s runtime  context
                                                             |  calls  
a procedure th3 instead of as a thread
                                                             |
                                                             v
                                                         proc_th3  
---  It can play in p3''s address space along with the other Th1. Th2  
can''t as it has just called proc th3 but after procedure return it can

Please note p3 ''s  critical region is properly protected and there is  
no reader / writer problem. Each thread runs at its on pace and  
eventually reports back to the snoring p3 using its conditional  
variable. This all works properly in the threading only environment  
and sometimes in mixed environment.

Th1,  th2, and proc th3 call a shared token dispenser as a reader  
only in process address space. This token dispenser is a C++  
template. The debugging bout is in how the thread''s register contexts  
and how they get set.
So far i''m at the fbt:genunix:segvn_faultpage:entry using dtracing  
stack and ustack functions. i''ll be writing more specific code to  
export the userland stack''s contexts.

My 1st week of Dtrace learning is fun and speaks highly of its  
capabilities and well thought out implementation! The `Solaris  
Performance and Tools` has jump started my investigation and highly  
recommend buying it along with  its companion `Solaris Internals`.

Questions: I''ve been to the wiki site and appreciate the maturing  
documentation but i''m a bit stumped regarding the below areas of  
exploration.
1) how do i set an absolute address probe in a pid provider of C++  
code? i figured out how to set a C++ function probe by using the  
mangled name. i get a compiler error on my attempt at an absolute  
address declaration in my attempts to figure out how to declare it.  
The address is part of the process space and not specific to eg  `main`.
2) can i declare multithreaded probes so that i can observe the  
specific probe firings per declared thread?
2.1) how do i set pid probes against future threads? how do i declare  
each thread''s run context in the provider''s part for the
potential
probe. these threads are not static but created and launched  
depending on the runtime context.
2.2) how do i get pid functions firing on these future threads?

I started with the dbx but it is not sensitive to the OS context  
switching of these threads. the combo dtrace / mdb is much better  
suited for me when i really get down and dirty after completing my  
above tracings.

I thank u for any suggestions
Dave Bone

Hi Dave,
> 1) how do i set an absolute address probe in a pid provider of C++  
> code? i figured out how to set a C++ function probe by using the  
> mangled name. i get a compiler error on my attempt at an absolute  
> address declaration in my attempts to figure out how to declare it.  
> The address is part of the process space and not specific to eg  `main`.
What are you trying and what''s the error you''re seeing?
> 2) can i declare multithreaded probes so that i can observe the  
> specific probe firings per declared thread?
I''m not sure exactly what you mean, but you can record your thread ID
when
a probe fires using the ''tid'' variable.
> 2.1) how do i set pid probes against future threads? how do i declare  
> each thread''s run context in the provider''s part for the
potential
> probe. these threads are not static but created and launched  
> depending on the runtime context.
> 2.2) how do i get pid functions firing on these future threads?
All threads in your process will be traced by the pid provider.

Adam

-- 
Adam Leventhal, Fishworks                        http://blogs.sun.com/ahl

Adam,
I appreciate your closing paragraph. As this is my very first time as  
Madona''s song goes to `open source community` dialogue, I assumed as  
u were the only one that responded to my first call-for-help that  
possibly my request was too esoteric and i did not want to make  
graffiti out of my request to the others. So i''ll also post the other  
private email with its attachments to the dtrace-discuss community  
after this email so they can bring-themselves-up-to-speed.

Concerning: Is __1cCbp6F_v_ (bp()) getting called in all threads that  
are spawned?
This is only called from main() once. The dynamic threads that are  
created afterwards have no knowledge of this global function.
The function was created as a starting point within the main function  
to lower the copious tracings for my bird-eye-view to my problem.

Concerning absolute addressing:
I still don''t know how to accomplish this though the Dtrace doc  
mentions that it can be done but without an example. I understand  
that a probe name left blank means trace the instructions. Using a  
module:function:offset makes the probe specific. My attempt is due to  
the mangled names of my c++ code and templates etc. Grabbing an  
absolute address from dbx out-of-the-debugger of sunstudio 12 seemed  
a lot easier than wrestling with the mangled name and possibly its  
mangled module etc.

Adam, how would u go about declaring such a pid probe given 0x12345  
as an absolute address within the process? Those were my attempts to  
get it going.

Concerning Attempt 3:  echo  
__1cGyacco2GParserOget_next_token6M_pn0AMCAbs_lr1_sym__::dis | mdb -p  
pid
> It looks like there''s likely no instruction at offset 0x291; how  
> did you
> get that address?
>
The address was obtained using dbx from sunstudio12. Please see the  
attachment `disassembly` showing the complete disassembled code for  
the get_next_token function. Below is the extracted disassembled  
instruction probed on.
__1cGyacco2GParserOget_next_token6M_pn0AMCAbs_lr1_sym__+0x291:
movl   $0x0,-0x24(%rbp)
Unfortunately the instruction exists. Here''s your request piped thru  
mdb.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: disassembly
Type: application/octet-stream
Size: 32772 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080129/5cdc1571/attachment.obj>
-------------- next part --------------

> Attempt 4: You''re instrumenting every offset probe in that  
> function. Are
> you sure that function is getting called?
>
Yup. The `4.log` attachment from previous email shows the threads  
being spawned  but no probe firings to the individual threads.
Using mdb there are 3 lwps 1 for the process and 2 spawned threads.

As this is my 2nd week at dtrace-mdb familiarization i now am madly  
reading while commuting by train the Solaris Internals book so that i  
can be more intimate to the lpw environment with stored regs. my gut  
feeling is the lwp 3''s registers take on  lpw 2 regs. I''m
hoping to
use Dtrace to refine my own debug prodding to conclude why the bug.

Dave


On 29-Jan-08, at 1:10 PM, Adam Leventhal wrote:
> On Mon, Jan 28, 2008 at 08:15:41PM -0500, david bone wrote:
>> I''ve attached 3 files:
>> 	1) o2debug1.d --- dtrace file showing my pid function traces and its
>> companion `4.log` file showing the output. I''ve commented
inside the
>> `d` file my objectives and observations. 	The `4.log` file shows the
>> threads spawned etc but no tracing of their functions.
>
> Is __1cCbp6F_v_ (bp()) getting called in all threads that are spawned?
>
>> 	2) o2debug2.d --- trying to set probe firing using absolute
>> addressing. the `d` file has comments and the Dtrace
compiler''s error
>> messages.
>
> Attempt 1: You''re trying to enable the 0x64e00c offset probe in  
> every function
> Attempt 2: You''re trying to use the address as the module;
that''s
> just wrong
> Attempt 3: Please send me the disassembly output for
> echo __1cGyacco2GParserOget_next_token6M_pn0AMCAbs_lr1_sym__::dis |  
> mdb -p pid
> It looks like there''s likely no instruction at offset 0x291; how  
> did you
> get that address?
> Attempt 4: You''re instrumenting every offset probe in that  
> function. Are
> you sure that function is getting called?
>
>> I thank u for your time.
>
> No problem. I''d appreciate it if you would post this sort of thing
> to the
> discussion list so that other people can benefit from the response and
> perhaps help you out.
>
> Adam
>
> -- 
> Adam Leventhal, Fishworks                        http:// 
> blogs.sun.com/ahl

On Tue, Jan 29, 2008 at 09:25:59PM -0500, david bone
wrote:
> Concerning: Is __1cCbp6F_v_ (bp()) getting called in all threads that  
> are spawned?
> This is only called from main() once. The dynamic threads that are  
> created afterwards have no knowledge of this global function.
> The function was created as a starting point within the main function  
> to lower the copious tracings for my bird-eye-view to my problem.
That''s why other threads aren''t showing up in your traced
view: you''re using
a thread local variable (self->...) as a predicate.
> Concerning absolute addressing:
> I still don''t know how to accomplish this though the Dtrace doc  
> mentions that it can be done but without an example. I understand  
> that a probe name left blank means trace the instructions. Using a  
> module:function:offset makes the probe specific. My attempt is due to  
> the mangled names of my c++ code and templates etc. Grabbing an  
> absolute address from dbx out-of-the-debugger of sunstudio 12 seemed  
> a lot easier than wrestling with the mangled name and possibly its  
> mangled module etc.
You can trace this by doing pid<pid>::-:<hex-address>. But be
careful! An
incorrectly aligned address can cause your application to crash.
> Adam, how would u go about declaring such a pid probe given 0x12345  
> as an absolute address within the process? Those were my attempts to  
> get it going.
> 
> Concerning Attempt 3:  echo  
> __1cGyacco2GParserOget_next_token6M_pn0AMCAbs_lr1_sym__::dis | mdb -p  
> pid
> >It looks like there''s likely no instruction at offset 0x291;
how
> >did you
> >get that address?
> >
> The address was obtained using dbx from sunstudio12. Please see the  
> attachment `disassembly` showing the complete disassembled code for  
> the get_next_token function. Below is the extracted disassembled  
> instruction probed on.
> __1cGyacco2GParserOget_next_token6M_pn0AMCAbs_lr1_sym__+0x291:
> movl   $0x0,-0x24(%rbp)
> Unfortunately the instruction exists. Here''s your request piped
thru
> mdb.
Curious. I''m surprised that it''s not letting you instrument
that
instruction. Show me this output:

dtrace -l -n
pid<pid>::__1cGyacco2GParserOget_next_token6M_pn0AMCAbs_lr1_sym__:
> Yup. The `4.log` attachment from previous email shows the threads  
> being spawned  but no probe firings to the individual threads.
> Using mdb there are 3 lwps 1 for the process and 2 spawned threads.
Again, this is because your program is not written to record the information
you apparently want to trace.
> As this is my 2nd week at dtrace-mdb familiarization i now am madly  
> reading while commuting by train the Solaris Internals book so that i  
> can be more intimate to the lpw environment with stored regs. my gut  
> feeling is the lwp 3''s registers take on  lpw 2 regs. I''m
hoping to
> use Dtrace to refine my own debug prodding to conclude why the bug.
You might want to take a look at this presentation:

  http://blogs.sun.com/ahl/entry/dtrace_boot_camp

- ahl

-- 
Adam Leventhal, Fishworks                        http://blogs.sun.com/ahl

Adam,
Thk u on the absolute address how-to for a pid probe, and link to refine my
Dtrace fluency.
>
>Curious. I''m surprised that it''s not letting you
instrument that
>instruction. Show me this output:
>
>dtrace -l -n
pid<pid>::__1cGyacco2GParserOget_next_token6M_pn0AMCAbs_lr1_sym__:
>
The attached file `disassembly2` contains the output showing the entry,return
probes with the instrumented instruction .
Dave
-------------- next part --------------
A non-text attachment was scrubbed...
Name: disassembly2
Type: application/octet-stream
Size: 32354 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080130/54156396/attachment.obj>

On Tue, Jan 29, 2008 at 09:25:59PM -0500, david bone
wrote:
> >It looks like there''s likely no instruction at offset 0x291;
how
> >did you
> >get that address?
> >
> The address was obtained using dbx from sunstudio12. Please see the  
> attachment `disassembly` showing the complete disassembled code for  
> the get_next_token function. Below is the extracted disassembled  
> instruction probed on.
> __1cGyacco2GParserOget_next_token6M_pn0AMCAbs_lr1_sym__+0x291:
> movl   $0x0,-0x24(%rbp)
> Unfortunately the instruction exists. Here''s your request piped
thru
> mdb.
The problem is that you need to specify the probe like this:

  pid$target::__1cGyacco2GParserOget_next_token6M_pn0AMCAbs_lr1_sym__:291

Adam

-- 
Adam Leventhal, Fishworks                        http://blogs.sun.com/ahl

Adam,
In conclusion to this thread:
1) Absolute addressing ---caution well noted when using it.
 The pid$target::-:<hex address> has Dtrace compile problems. The Dtrace
compiler output: Failed to compile script o2debug4.d: line 62: extraneous
argument ''9364'' ($1 is not referenced). As I can work around
using offsets I do not want to waste your time on this one.
2) my stupidity on local verses global predicate variables to fire all threads
within the pid --- thk u for the lesson
3) as below, my learning refinement on how-to-play-with defining offsets to get
them to fire. noting your debugging approach allows me to home the offset
definition.

Now i can get them to fire under my control to refine the bug-watch.
Thk u for your help and thread closed:}
Sincerely
Dave


On Wednesday, January 30, 2008, at 10:11AM, "Adam Leventhal" <ahl
at eng.sun.com> wrote:
>On Tue, Jan 29, 2008 at 09:25:59PM -0500, david bone wrote:
>> >It looks like there''s likely no instruction at offset
0x291; how
>> >did you
>> >get that address?
>> >
>> The address was obtained using dbx from sunstudio12. Please see the  
>> attachment `disassembly` showing the complete disassembled code for  
>> the get_next_token function. Below is the extracted disassembled  
>> instruction probed on.
>> __1cGyacco2GParserOget_next_token6M_pn0AMCAbs_lr1_sym__+0x291:
>> movl   $0x0,-0x24(%rbp)
>> Unfortunately the instruction exists. Here''s your request
piped thru
>> mdb.
>
>The problem is that you need to specify the probe like this:
>
>  pid$target::__1cGyacco2GParserOget_next_token6M_pn0AMCAbs_lr1_sym__:291
>
>Adam
>
>-- 
>Adam Leventhal, Fishworks                        http://blogs.sun.com/ahl
>
>