Can someone, more learned in the ways of dtrace point me at what to look at to help understand why the kernel on one machine is using tons of memory, while another machine doing the same task/same user load is not. swapinfo for the "afflicted" machine shows RAM _______Total 16384 Mb RAM Unusable 73 Mb RAM Kernel 9226 Mb RAM Locked 2 Mb RAM Used 2692 Mb RAM Avail 4389 Mb While the second "unsick" machine shows RAM _______Total 16384 Mb RAM Unusable 73 Mb RAM Kernel 3319 Mb RAM Locked 2 Mb RAM Used 2383 Mb RAM Avail 10604 Mb The first machine is using about 6G of memory more than the second. What providers and or values should I be looking at to get a better picture of what is going on? and/or what parts of the kernel are using the memory? Thanks in advance Jeff This message posted from opensolaris.org
On Thu, 6 Apr 2006, Jeffrey Collyer wrote:> Can someone, more learned in the ways of dtrace point me at what to look at to help understand why the kernel on one machine is using tons of memory, while another machine doing the same task/same user load is not. > > swapinfo for the "afflicted" machine shows > > RAM _______Total 16384 Mb > RAM Unusable 73 Mb > RAM Kernel 9226 Mb > RAM Locked 2 Mb > RAM Used 2692 Mb > RAM Avail 4389 Mb > > While the second "unsick" machine shows > > RAM _______Total 16384 Mb > RAM Unusable 73 Mb > RAM Kernel 3319 Mb > RAM Locked 2 Mb > RAM Used 2383 Mb > RAM Avail 10604 Mb > > The first machine is using about 6G of memory more than the second. > > What providers and or values should I be looking at to get a better picture of what is going on? > and/or what parts of the kernel are using the memory?Hi Jeffrey, I typically use the mdb kmastat dcmd to view kernel memory usage: $ mdb -k> ::memstat[ ... ]> ::kmastat[ ... ] The kernel engineers might have some additional guidance. Hope this helps, - Ryan -- UNIX Administrator http://daemons.net/~matty
Matty wrote: ...> I typically use the mdb kmastat dcmd to view kernel memory usage: > > $ mdb -k > >> ::memstat > [ ... ] > >> ::kmastat > [ ... ] > > The kernel engineers might have some additional guidance.Yep ::memstat and ::kmastat are indeed the best tools to figure this out. Would you happen to be running ZFS on the "sick" machine by chance? - Eric
G''Day, On Thu, 6 Apr 2006, Eric Lowe wrote:> Matty wrote: > ... > > I typically use the mdb kmastat dcmd to view kernel memory usage: > > > > $ mdb -k > > > >> ::memstat > > [ ... ] > > > >> ::kmastat > > [ ... ] > > > > The kernel engineers might have some additional guidance. > > Yep ::memstat and ::kmastat are indeed the best tools to figure this out.In fact, swapinfo has one of the longest "SEE ALSO" sections I''ve written, # more swapinfo.d ... * SEE ALSO: swapinfo - K9Toolkit, http://www.brendangregg.com/k9toolkit.html * vmstat 1 2; swap -s; echo ::memstat | mdb -k * RMCmem - The MemTool Package * RICHPse - The SE Toolkit * "Clearing up swap space confusion" Unix Insider, Adrian Cockcroft * "Solaris Internals", Jim Mauro, Richard McDougall * /usr/include/vm/anon.h, /usr/include/sys/systm.h ... I''m going to revisit the Perl Sun::Solaris::Kstat version - I''d love to be able to dish this info out from kstat as non-root. cheers, Brendan
Brendan Gregg wrote:>>> The kernel engineers might have some additional guidance. >> Yep ::memstat and ::kmastat are indeed the best tools to figure this out. > > In fact, swapinfo has one of the longest "SEE ALSO" sections I''ve written, > > # more swapinfo.d > ... > * SEE ALSO: swapinfo - K9Toolkit, http://www.brendangregg.com/k9toolkit.html > * vmstat 1 2; swap -s; echo ::memstat | mdb -k > * RMCmem - The MemTool Package > * RICHPse - The SE Toolkit > * "Clearing up swap space confusion" Unix Insider, Adrian Cockcroft > * "Solaris Internals", Jim Mauro, Richard McDougall > * /usr/include/vm/anon.h, /usr/include/sys/systm.h > ... > > I''m going to revisit the Perl Sun::Solaris::Kstat version - I''d love to be > able to dish this info out from kstat as non-root.If there is anything you want out of the VM system which isn''t being exported the way you want, get with me offline and let''s get those RFEs filed and fixed. Observability is bubbling up to the top of my list for VM; we can''t design a good new VM system if we can''t understand the current one! :) Cheers - Eric