dtrace discuss - Sep 2007 - Any debugging pragmas to help with "failed to grab pid xxx" ?

I have a dtrace script which works OK when grabbing a single pid:
pid$1:*::entry
{ 
  self->depth++;
}

But if I extend it to try for 6 pids concurrently, it fails on the forth pid:
pid$1:*::entry, pid$2:*::entry, pid$3:*::entry, pid$4:*::entry, pid$5:*::entry,
pid$6:*::entry
{ 
  self->depth++;
}

To wit,
dtrace: failed to compile script /opt/utils/libtimesMulti.d: failed to grab pid
23409: unanticipated system error

Any thoughts? This is Solaris 10U2, aka 0606. Thanks in advance for any tips.


--
This message posted from opensolaris.org

Hey David,

This looks a lot like the question you asked on July 19th of this year. We
determined that the problem then was caused by libproc exhausting the file
descriptor limit. Let us know if you already eliminated that cause.

  http://www.opensolaris.org/jive/thread.jspa?messageID=138829

The related bug has still not been fixed (any takers?)
  6277664 Pgrab() failures due to the fd rlimit are not well reported

Adam

On Wed, Sep 26, 2007 at 02:23:00PM -0700, David McDaniel
wrote:
> I have a dtrace script which works OK when grabbing a single pid:
> pid$1:*::entry
> { 
>   self->depth++;
> }
> 
> But if I extend it to try for 6 pids concurrently, it fails on the forth
pid:
> pid$1:*::entry, pid$2:*::entry, pid$3:*::entry, pid$4:*::entry,
pid$5:*::entry, pid$6:*::entry
> { 
>   self->depth++;
> }
> 
> To wit,
> dtrace: failed to compile script /opt/utils/libtimesMulti.d: failed to grab
pid 23409: unanticipated system error
> 
> Any thoughts? This is Solaris 10U2, aka 0606. Thanks in advance for any
tips.
> 
> 
> --
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl

On 9/26/07, Adam Leventhal <ahl at eng.sun.com>
wrote:
> Hey David,
>
> This looks a lot like the question you asked on July 19th of this year. We
> determined that the problem then was caused by libproc exhausting the file
> descriptor limit. Let us know if you already eliminated that cause.
>
>   http://www.opensolaris.org/jive/thread.jspa?messageID=138829
>
> The related bug has still not been fixed (any takers?)
>   6277664 Pgrab() failures due to the fd rlimit are not well reported
I''ve put a fix for this bug up at
http://interstel.net/~mynhier/6277664/, if anyone would care to do a
code review, and if someone would care to sponsor this.  (Or I can
request a sponsor via the normal route.)

I''ve tested this with a simple program that first exhausts file
descriptors before calling Pgrab()
(http://interstel.net/~mynhier/6277664/pgraberr.c):

mynhierc at betasx6.nyc:/var/tmp/chadfoo> ./pgraberr 3111
Pgrab failed: unanticipated system error
mynhierc at betasx6.nyc:/var/tmp/chadfoo> env
LD_PRELOAD=/fs/opensolaris/6277664/usr/src/lib/libproc/i386/libproc.so.1
./pgraberr 3111
Pgrab failed: too many open files
mynhierc at betasx6.nyc:/var/tmp/chadfoo>

Chad Mynhier

> I''ve put a fix for this bug up at
> http://interstel.net/~mynhier/6277664/, if anyone would care to do a
> code review, and if someone would care to sponsor this.  (Or I can
> request a sponsor via the normal route.)
Hey Chad,

This is looking good. I''d be happy to sponsor this. Let''s
finalize this
offline.

Adam

-- 
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl

Doh! Of course its the same. Thats the good news. 
The bad news is that now after spinning around for a few seconds, it fails:
thing.d  : line28  :  failed to create probe in process 1330: Not enough space

  I flailed around with some of the buf/agg/dynvar size thingies, etc, but since
I dont really know what resource is insufficient I dont know if there is
anything I can do about it.

regards
-d


--
This message posted from opensolaris.org

> The bad news is that now after spinning around for a few seconds, it fails:
> thing.d  : line28  :  failed to create probe in process 1330: Not enough
space
Since you''re trying to enable a ton of probes you may actually be
running out of pid probes. You can set the fasttrap-max-probes in
/kernel/drf/fasttrap.conf to a higher value. There''s entry on the
DTrace wiki about this which may be helpful:

 
http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Errors#failed_to_create_probe_..._Not_enough_space

Adam

-- 
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl