dtrace discuss - Jul 2010 - Invalid address in call to ustack()?

Hi all,

I''m trying to use the following profile probe to grab stack traces at 
regular intervals:
> profile-$FREQ /pid==$PID/ {
>         @profile[ustack()]=count;
> }
and am getting quite a few errors like the following:
> dtrace: error on enabled probe ID 4 (ID 60477: 
> profile:::profile-7777us): invalid address (0x100000) in action #2
> dtrace: error on enabled probe ID 4 (ID 60477: 
> profile:::profile-7777us): invalid address (0x100000) in action #2
> dtrace: error on enabled probe ID 4 (ID 60477: 
> profile:::profile-7777us): invalid address (0x100000) in action #2
> dtrace: error on enabled probe ID 4 (ID 60477: 
> profile:::profile-7777us): invalid address (0x17f3fa000) in action #2
> dtrace: error on enabled probe ID 4 (ID 60477: 
> profile:::profile-7777us): invalid address (0x100000) in action #2
> dtrace: error on enabled probe ID 4 (ID 60477: 
> profile:::profile-7777us): invalid address (0x0) in action #2
> dtrace: error on enabled probe ID 4 (ID 60477: 
> profile:::profile-7777us): invalid address (0x100000) in action #2
There''s enough of those appearing that I suspect it''s
increasing
dtrace''s probe effect. I''m tempted to just throw a -q at it,
that would
only mask the symptoms. Is there something else I should do to prevent 
the errors from occurring at all?

I''m on Solaris 10/sparcv9, and the app being traced is heavily
threaded,
in case that makes any difference.

Thanks!
Ryan

Hey Ryan,
> There''s enough of those appearing that I suspect it''s
increasing dtrace''s probe effect. I''m tempted to just throw a
-q at it, that would only mask the symptoms. Is there something else I should do
to prevent the errors from occurring at all?

It''s hard to say why you might be hitting that. You could investigate
by doing something like this:

ERROR
{
	stop();
	printf("stopped %d due to an error", pid);
	exit();
}

Then you can use gcore <pid> to grab a core dump and prun <pid> to
set the process running again. From that core we should be able to figure out
why the stack backtrace failed.
> I''m on Solaris 10/sparcv9, and the app being traced is heavily
threaded, in case that makes any difference.

Threading should not be a contributing factor to this issue.

Adam

On Jul 10, 2010, at 12:52 AM, Ryan Johnson wrote:
> Hi all,
> 
> I''m trying to use the following profile probe to grab stack traces
at regular intervals:
>> profile-$FREQ /pid==$PID/ {
>>        @profile[ustack()]=count;
>> }
> 
> and am getting quite a few errors like the following:
>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x100000) in action #2
>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x100000) in action #2
>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x100000) in action #2
>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x17f3fa000) in action #2
>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x100000) in action #2
>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x0) in action #2
>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x100000) in action #2
> 
> There''s enough of those appearing that I suspect it''s
increasing dtrace''s probe effect. I''m tempted to just throw a
-q at it, that would only mask the symptoms. Is there something else I should do
to prevent the errors from occurring at all?
> 
> I''m on Solaris 10/sparcv9, and the app being traced is heavily
threaded, in case that makes any difference.
> 
> Thanks!
> Ryan
> 
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

--
Adam Leventhal, Fishworks                        http://blogs.sun.com/ahl

*** Migrating this back to the list (curse the broken reply-to headers...)

Hi Jim,

Thanks for pointing that out -- I''d never noticed the section about 
profile probe args on the wiki before. However, I''ve never seen
anything
which suggests [u|k]stack depend on any of the arg*.

Adding the arg1 predicate reduces the number of errors, but comparing 
runs using /arg1/ vs /!arg1/ shows that most errors actually occur with 
/arg1/, even though I''m hitting both cases with roughly equal frequency
(241k vs 216k samples over 10s)

Regards,
Ryan

On 7/10/2010 4:37 PM, Jim Mauro wrote:
> Try adding arg1 to the predicate;
>
> / arg1&&  pid == $PID /
>
>
> On Jul 10, 2010, at 3:52 AM, Ryan Johnson wrote:
>
>    
>> Hi all,
>>
>> I''m trying to use the following profile probe to grab stack
traces at regular intervals:
>>      
>>> profile-$FREQ /pid==$PID/ {
>>>         @profile[ustack()]=count;
>>> }
>>>        
>> and am getting quite a few errors like the following:
>>      
>>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x100000) in action #2
>>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x100000) in action #2
>>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x100000) in action #2
>>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x17f3fa000) in action #2
>>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x100000) in action #2
>>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x0) in action #2
>>> dtrace: error on enabled probe ID 4 (ID 60477:
profile:::profile-7777us): invalid address (0x100000) in action #2
>>>        
>> There''s enough of those appearing that I suspect it''s
increasing dtrace''s probe effect. I''m tempted to just throw a
-q at it, that would only mask the symptoms. Is there something else I should do
to prevent the errors from occurring at all?
>>
>> I''m on Solaris 10/sparcv9, and the app being traced is heavily
threaded, in case that makes any difference.
>>
>> Thanks!
>> Ryan
>>
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
>>      
>

On 7/10/2010 10:13 PM, Adam Leventhal wrote:
>> There''s enough of those appearing that I suspect it''s
increasing dtrace''s probe effect. I''m tempted to just throw a
-q at it, that would only mask the symptoms. Is there something else I should do
to prevent the errors from occurring at all?
>>      
> It''s hard to say why you might be hitting that. You could
investigate by doing something like this:
>
> ERROR
> {
> 	stop();
> 	printf("stopped %d due to an error", pid);
> 	exit();
> }
>
> Then you can use gcore<pid>  to grab a core dump and prun<pid> 
to set the process running again. From that core we should be able to figure out
why the stack backtrace failed.
>    
OK, I did that (took a looong time -- 5GB dump). I don''t see anything 
out of the ordinary. Also tried attaching directly with dbx and pstack 
-- no errors. I added a printout of the offending tid in hopes of 
narrowing things down, but still nothing jumps out -- random tid and 
function every time.

Nearly always the offending address is 0x0, but I did (once) manage to 
get 0x100000; the output of pstack never shows any sign of the bad 
address, though. It always goes all the way down to _lwp_start.

Disassembling functions showed nothing useful either -- the errors seem 
to come after any instruction (even those which do not reference memory, 
like "rd        %pc, %o7").

Is there something specific I should look for?

Thanks!
Ryan