dtrace discuss - Oct 2008 - Why does not always the ustack() output the function name?

Hello, super dtrace masters,

When I use Dscripts in which ustack() is involved, 
it sometimes output the stacktrace with their names of the functions,
but sometime not.

I''m confused because of these inconsistent results.
Does anyone know this reason?

Thanks in advance.
--
This message posted from opensolaris.org

Yasufumi CHIDA wrote:
> Hello, super dtrace masters,
> 
> When I use Dscripts in which ustack() is involved, 
> it sometimes output the stacktrace with their names of the functions,
> but sometime not.
> 
> I''m confused because of these inconsistent results.
> Does anyone know this reason?
most likely, you''re seeing the effects of aynchronous post-processing
of an
event: data is recorded when the probe fires, but only post-processed 
(converting adresses to names, among other things) later. If the process 
under inspection has gone away by the time post-processing happens,
there''s
nothing to get the names from, and all you see is addresses.

HTH
Michael
-- 
Michael Schuster     http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''

On Mon, Oct 20, 2008 at 11:39 PM, michael schuster
<Michael.Schuster at sun.com> wrote:
> Yasufumi CHIDA wrote:
>> Hello, super dtrace masters,
>>
>> When I use Dscripts in which ustack() is involved,
>> it sometimes output the stacktrace with their names of the functions,
>> but sometime not.
>>
>> I''m confused because of these inconsistent results.
>> Does anyone know this reason?
>
> most likely, you''re seeing the effects of aynchronous
post-processing of an
> event: data is recorded when the probe fires, but only post-processed
> (converting adresses to names, among other things) later. If the process
> under inspection has gone away by the time post-processing happens,
there''s
> nothing to get the names from, and all you see is addresses.
Actually, could you give a little more detail?  (Yasufumi, not Michael.)

Specifically, are you getting inconsistent results from the same
process or program?  Or are you getting inconsistent results from
different processes/programs?  If it''s the former, then
Michael''s
answer makes sense.  If it''s the latter, then it may be simpler than
that -- some of the processes may be stripped, in which case you''d
have no symbol data to use for translating function addresses.

Chad

Hello, Chad,

Thank you for getting interested in detail.
In my case, "inconsistent results from different program" was that.

However, dtrace ustack() showed results from some programs with the name of
"_start()",  "main()",
"libc.so.1''_open()" and so on,
but from the other programs, without name of functions but hexa-decimal
addresses.


e.g.

(with name)

bash setpgrp 13
libc.so.1`_syscall6+0x1c
35c6c
34fa8
bash`execute_command_internal+0x414
bash`execute_command+0x50
bash`reader_loop+0x220
bash`main+0x90c
bash`_start+0x108




(without name)  ... another Dscript
''/usr/openwin/lib/X11/XtErrorDB'' failed
febbcf78
febb05a0
fec97b38
fec97a78
fedbbffc
fedbbeac
fedbbe40
fedc0220
fedc037c
fed8fb6c
fed8f2f8
fed8f290
cf3f8
3f648
d1c98
5c658




That ''s inconsistency I said. I still have no idea...

Yasufumi
--
This message posted from opensolaris.org

>Hello, Chad,
>
>Thank you for getting interested in detail.
>In my case, "inconsistent results from different program" was
that.
>
>However, dtrace ustack() showed results from some programs with the name of
"_start()",  "main()",
 "libc.so.1''_open()" and so on, 
>but from the other programs, without name of functions but hexa-decimal
addresses.
>
>
dtrace needs to be able open the process; if the process has terminated,
all you get are hexidecimal addresses.

DTrace is async: first it generates the probes output in the probe
context (sync); then later it pretty prints the output, outside of the probe
context (after the fact). 

Casper

On Oct 23, 2008, at 2:20 AM, Casper.Dik at Sun.COM
wrote:
>> Thank you for getting interested in detail.
>> In my case, "inconsistent results from different program" was
that.
>>
>> However, dtrace ustack() showed results from some programs with the  
>> name of "_start()",  "main()",
> "libc.so.1''_open()" and so on,
>> but from the other programs, without name of functions but hexa- 
>> decimal addresses.
>>
>
> dtrace needs to be able open the process; if the process has  
> terminated,
> all you get are hexidecimal addresses.
>
> DTrace is async: first it generates the probes output in the probe
> context (sync); then later it pretty prints the output, outside of  
> the probe
> context (after the fact).

To be a little clearer, the ustack() action records addresses when the  
probe
fires, but those addresses aren''t converted to symbols until later in  
user-
land, and at that point the process from which the stack came may have  
gone
away. Here''s how I solve this problem:

<some probe where you take a ustack()>
{
	ustack();
	procs[pid] = 1;
}

syscall::rexit:entry
/procs[pid]/
{
	procs[pid] = 0;
	stop();
	system("prun %d", pid);
}

What this does is catches processes as they call the exit system call  
and
forces them to pause long enough for us to gather ustack() data. This  
isn''t
fool proof since applications that exit abnormally will escape.

Adam

--
Adam Leventhal, Fishworks                        http://blogs.sun.com/ahl