dtrace discuss - Mar 2008 - dtrace: failed to grab pid 16537: process is traced

I''m getting an odd behavior while trying to trace a mysql process using
the -p flag on a Sun Cluster.

This fails

# dtrace -n ''pid$target::*mysql_parse*:entry {}'' -p `pgrep -x
mysqld`
dtrace: failed to grab pid 16537: process is traced

Yet this works
# dtrace -n ''pid$1::*mysql_parse*:entry {}'' `pgrep -x mysqld`
dtrace: description ''pid$1::*mysql_parse*:entry '' matched 1
probe

All processes managed within the Sun Cluster agent framework appear to 
exhibit this behavior yet if they aren''t managed by cluster they appear
to work.  Has anyone seen this problem? Also any pointers to investigate 
this further?

# uname -a
SunOS wolverine 5.10 Generic_118833-36 sun4u sparc SUNW,Sun-Fire


Thanks

Dave

On Fri, Mar 21, 2008 at 01:01:14PM -0500, David Blasingame, Area Systems Support
Engineer wrote:
> I''m getting an odd behavior while trying to trace a mysql process
using
> the -p flag on a Sun Cluster.
> 
> This fails
> 
> # dtrace -n ''pid$target::*mysql_parse*:entry {}'' -p
`pgrep -x mysqld`
> dtrace: failed to grab pid 16537: process is traced
> 
> Yet this works
> # dtrace -n ''pid$1::*mysql_parse*:entry {}'' `pgrep -x
mysqld`
> dtrace: description ''pid$1::*mysql_parse*:entry '' matched
1 probe
> 
> All processes managed within the Sun Cluster agent framework appear to 
> exhibit this behavior yet if they aren''t managed by cluster they
appear
> to work.  Has anyone seen this problem? Also any pointers to investigate 
> this further?
> 
> # uname -a
> SunOS wolverine 5.10 Generic_118833-36 sun4u sparc SUNW,Sun-Fire
> 
> 
> Thanks
> 
> Dave
Sun Cluster has a long-standing design limitation/flaw that it uses /proc
to trace processes to effect their restart.  This interferes with the
ability of other debuggers like dbx, DTrace, and proc(1) tools to work on them.
It is for this reason that when we invented SMF in Solaris 10, we implemented
a proper kernel mechanism for managing process restart, called contracts,
independent of /proc, to handle notification events etc. for restart.
Sun Cluster needs to be converted to use it.  Hopefully someone from
that team can comment on the software release schedule for that change.

-Mike

-- 
Mike Shapiro, Sun Microsystems Fishworks. blogs.sun.com/mws/