dtrace discuss - Mar 2006 - problem using dtrace to find memory consumers

Hi All, 

I''m in the middle of trying to debug an issue where the kernel appears
to allocate most of memory while running a particular workload in the
kmem_oversize_arena.  I created the script at the end, started the script,
redirecting output to a file, started off my workload, the system got into the
low memory state where it appears hung so I dropped to obp and did a sync to get
a corefile.  The file with the dtrace output shows no problems as freemem is
large when all of the recorded probes fired.  It looks like what happened is
that dtrace got turned into a zombie:
> ::ps ! grep dtrace
Z  23293    244    244 1048576      0 0x4a004002 0000060001ee17f8 dtrace

::dtrace_state also returns nothing so clearly I did not capture the data that I
wanted to get, mainly the last number of consumers who really drove the memory
usage in the system down.

So, I''m looking for ideas on how to prevent this from happening, as
I''d have to guess that dtrace tried to allocate some memory somewhere,
couldn''t do it as there was none to be had and thus died.

Is there any "secret flag" which can be used to tell dtrace to lock
all of it''s memory so that once it''s memory is allocated, it
will be locked (basically memcntl(MLOCK_AS))?  This should help, but possibly
not eliminate, the problem.

I can also put it in the realtime class to help out it''s priority, but
this may not be enough.

Would using anonymous tracing allow this to work as I could always extract the
data from the running system as well as from the core file itself?  Assuming
that the buffers were large enough, I should hopefully be able to see freemem
taking a dive as large buffers get allocated.

Any suggestions would be greatly appreciated.

Thanks,

Mike

cat oversize.d
#!/usr/sbin/dtrace -Cs

::vmem_alloc:entry
/ args[0] == `kmem_oversize_arena /
{
        self->traceme = 1;
        trace(`freemem);
        trace(args[1]);
        trace(args[2]);
        trace(execname);
        trace(curthread);
        stack(20);
}

::vmem_alloc:return
/ self->traceme /
{
        trace(args[1]);
        trace(execname);
        trace(curthread);
        self->traceme = 0;
}

::vmem_free:entry
/ args[0] == `kmem_oversize_arena /
{
        trace(`freemem);
        trace(args[1]);
        trace(args[2]);
        trace(execname);
        trace(curthread);
        stack(20);
}
This message posted from opensolaris.org

> I''m in the middle of trying to debug an issue where the kernel
appears to allocate most of memory while running a particular workload in the
kmem_oversize_arena.  I created the script at the end, started the script,
redirecting output to a file, started off my workload, the system got into the
low memory state where it appears hung so I dropped to obp and did a sync to get
a corefile.  The file with the dtrace output shows no problems as freemem is
large when all of the recorded probes fired.  It looks like what happened is
that dtrace got turned into a zombie:
> 
> > ::ps ! grep dtrace
> Z  23293    244    244 1048576      0 0x4a004002 0000060001ee17f8 dtrace
> 
> ::dtrace_state also returns nothing so clearly I did not capture the data
that I wanted to get, mainly the last number of consumers who really drove the
memory usage in the system down.
> 
> So, I''m looking for ideas on how to prevent this from happening,
as I''d have to guess that dtrace tried to allocate some memory
somewhere, couldn''t do it as there was none to be had and thus died.
That seems likely -- but note that the memory allocation failure was from
dtrace(1M), not dtrace(7D).  To achieve what you''re after, use ring
buffering.
(You can use ring buffering _and_ anonymous tracing, but that probably
won''t
be necessary.)  This is discussed in both the "Postmortem Tracing" and
"Buffers and Buffering" chapters of the documentation...

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc