G''Day Folks, While I have a few minutes online (which is rare at the moment - travel), let me post about a couple of new scripts. I''ve just uploaded them to http://www.brendangregg.com/dtrace.html. They are to profile user-level functions and libaries; and kernel-level functions and modules. They sample, rather than trace (which can be both good and bad). Here is hotuser sampling functions from a gunzip command, # ./hotuser -c ''gunzip contents.gz'' Sampling... Hit Ctrl-C to end. FUNCTION COUNT PCNT libc.so.1`_free_unlocked 1 0.1% gunzip`unzip 1 0.1% ld.so.1`strcmp 1 0.1% gunzip`inflate_dynamic 1 0.1% libc.so.1`_write 1 0.1% gunzip`write_buf 1 0.1% gunzip`0x2d990 2 0.3% libc.so.1`write 2 0.3% gunzip`0x2d994 2 0.3% ld.so.1`rtld_db_preinit 3 0.4% gunzip`0x2d98c 7 0.9% gunzip`huft_build 9 1.2% libc_psr.so.1`memcpy 138 18.5% gunzip`inflate_codes 233 31.2% gunzip`updcrc 344 46.1% This reports that 344 samples found updcrc() on the CPU, which was 46.1% of the samples for this PID. This is likely to be the hottest function in that process. It isn''t easy to get this information from other scripts I have published, such as dappprof or dapptrace, which carefully trace data. Perhaps these sampling scripts may prove more practical. Libaries can also sampled, rather than functions, # ./hotuser -lc ''grep brendan contents'' Sampling... Hit Ctrl-C to end. LIBRARY COUNT PCNT ld.so.1 3 0.7% libc_psr.so.1 3 0.7% grep 23 5.1% libc.so.1 109 24.3% libgen.so.1 310 69.2% This shows that instructions from libgen.so.1 was on-CPU during 310 samples. A companion tool exists, hotkernel, for doing the same profiling for the kernel. Since they are both new tools, I''m still testing and contemplating their accuracy. ... Tool internals These work by aggregating on addresses, then printing out the aggregate using either %a or %A. Then, Perl is used to post process the output - in particular to drop the instruction offset so that we have function names, or to drop more for libraries/modules. Yes - I seem to recall Bryan mentioning that dropping the instruction offset was going to be a new DTrace capability, so future versions of hotuser/hotkernel should run the new DTrace feature if available. I originally wrote these tools a long time ago, and was never satisified with Perl post processing (I''m sure many of you have thought of writing these tools as well). Nor did I find a suitable DTrace hack to make it work efficiently from pure DTrace. Since the output is rather useful, I''ll bite the bullet and go with Perl post-processing for now... ... enjoy! Brendan
Hi Brendan, I noticed that you have to eat the command''s STDOUT in hotuser. I tried "hotuser -c ''ls > stuff''", which I assume didn''t work because dtrace doesn''t use a shell to invoked the target. How would I separate my STDOUT, so that it doesn''t get mingled with yours? Is this a good reason for having the ability to direct trace output to a different file descriptor? Thanks, Chip Brendan Gregg wrote:>G''Day Folks, > >While I have a few minutes online (which is rare at the moment - travel), >let me post about a couple of new scripts. I''ve just uploaded them to >http://www.brendangregg.com/dtrace.html. > > >They are to profile user-level functions and libaries; and kernel-level >functions and modules. They sample, rather than trace (which can be both >good and bad). Here is hotuser sampling functions from a gunzip command, > > # ./hotuser -c ''gunzip contents.gz'' > Sampling... Hit Ctrl-C to end. > > FUNCTION COUNT PCNT > libc.so.1`_free_unlocked 1 0.1% > gunzip`unzip 1 0.1% > ld.so.1`strcmp 1 0.1% > gunzip`inflate_dynamic 1 0.1% > libc.so.1`_write 1 0.1% > gunzip`write_buf 1 0.1% > gunzip`0x2d990 2 0.3% > libc.so.1`write 2 0.3% > gunzip`0x2d994 2 0.3% > ld.so.1`rtld_db_preinit 3 0.4% > gunzip`0x2d98c 7 0.9% > gunzip`huft_build 9 1.2% > libc_psr.so.1`memcpy 138 18.5% > gunzip`inflate_codes 233 31.2% > gunzip`updcrc 344 46.1% > >This reports that 344 samples found updcrc() on the CPU, which was 46.1% >of the samples for this PID. This is likely to be the hottest function in >that process. > >It isn''t easy to get this information from other scripts I have published, >such as dappprof or dapptrace, which carefully trace data. Perhaps >these sampling scripts may prove more practical. > > >Libaries can also sampled, rather than functions, > > # ./hotuser -lc ''grep brendan contents'' > Sampling... Hit Ctrl-C to end. > > LIBRARY COUNT PCNT > ld.so.1 3 0.7% > libc_psr.so.1 3 0.7% > grep 23 5.1% > libc.so.1 109 24.3% > libgen.so.1 310 69.2% > >This shows that instructions from libgen.so.1 was on-CPU during 310 >samples. > >A companion tool exists, hotkernel, for doing the same profiling for the >kernel. Since they are both new tools, I''m still testing and contemplating >their accuracy. > >... > >Tool internals > >These work by aggregating on addresses, then printing out the aggregate >using either %a or %A. Then, Perl is used to post process the output - in >particular to drop the instruction offset so that we have function names, >or to drop more for libraries/modules. > >Yes - I seem to recall Bryan mentioning that dropping the instruction >offset was going to be a new DTrace capability, so future versions of >hotuser/hotkernel should run the new DTrace feature if available. > >I originally wrote these tools a long time ago, and was never satisified >with Perl post processing (I''m sure many of you have thought of >writing these tools as well). Nor did I find a suitable DTrace hack >to make it work efficiently from pure DTrace. Since the output is rather >useful, I''ll bite the bullet and go with Perl post-processing for now... > >... > >enjoy! > >Brendan > > > >_______________________________________________ >dtrace-discuss mailing list >dtrace-discuss at opensolaris.org > >
G''Day Chip, On Sat, 1 Jul 2006, Chip Bennett wrote:> Hi Brendan, > > I noticed that you have to eat the command''s STDOUT in hotuser. I tried > "hotuser -c ''ls > stuff''", which I assume didn''t work because dtrace > doesn''t use a shell to invoked the target. How would I separate my > STDOUT, so that it doesn''t get mingled with yours? Is this a good > reason for having the ability to direct trace output to a different file > descriptor?Yes - I considered printing the command''s STDOUT but didn''t think it was important. You are right, I should do something with it; such as print it and provide a -q option to not print it. Using shell metacharacters within a dtrace -c won''t work, as dtrace doesn''t invoke a sub shell. There is a way to redirect the dtrace probe output to a different file descriptor: eg -o /dev/fd/2. cheers, Brendan> Thanks, > Chip > > Brendan Gregg wrote: > > >G''Day Folks, > > > >While I have a few minutes online (which is rare at the moment - travel), > >let me post about a couple of new scripts. I''ve just uploaded them to > >http://www.brendangregg.com/dtrace.html. > > > > > >They are to profile user-level functions and libaries; and kernel-level > >functions and modules. They sample, rather than trace (which can be both > >good and bad). Here is hotuser sampling functions from a gunzip command, > > > > # ./hotuser -c ''gunzip contents.gz'' > > Sampling... Hit Ctrl-C to end. > > > > FUNCTION COUNT PCNT > > libc.so.1`_free_unlocked 1 0.1% > > gunzip`unzip 1 0.1% > > ld.so.1`strcmp 1 0.1% > > gunzip`inflate_dynamic 1 0.1% > > libc.so.1`_write 1 0.1% > > gunzip`write_buf 1 0.1% > > gunzip`0x2d990 2 0.3% > > libc.so.1`write 2 0.3% > > gunzip`0x2d994 2 0.3% > > ld.so.1`rtld_db_preinit 3 0.4% > > gunzip`0x2d98c 7 0.9% > > gunzip`huft_build 9 1.2% > > libc_psr.so.1`memcpy 138 18.5% > > gunzip`inflate_codes 233 31.2% > > gunzip`updcrc 344 46.1% > > > >This reports that 344 samples found updcrc() on the CPU, which was 46.1% > >of the samples for this PID. This is likely to be the hottest function in > >that process. > > > >It isn''t easy to get this information from other scripts I have published, > >such as dappprof or dapptrace, which carefully trace data. Perhaps > >these sampling scripts may prove more practical. > > > > > >Libaries can also sampled, rather than functions, > > > > # ./hotuser -lc ''grep brendan contents'' > > Sampling... Hit Ctrl-C to end. > > > > LIBRARY COUNT PCNT > > ld.so.1 3 0.7% > > libc_psr.so.1 3 0.7% > > grep 23 5.1% > > libc.so.1 109 24.3% > > libgen.so.1 310 69.2% > > > >This shows that instructions from libgen.so.1 was on-CPU during 310 > >samples. > > > >A companion tool exists, hotkernel, for doing the same profiling for the > >kernel. Since they are both new tools, I''m still testing and contemplating > >their accuracy. > > > >... > > > >Tool internals > > > >These work by aggregating on addresses, then printing out the aggregate > >using either %a or %A. Then, Perl is used to post process the output - in > >particular to drop the instruction offset so that we have function names, > >or to drop more for libraries/modules. > > > >Yes - I seem to recall Bryan mentioning that dropping the instruction > >offset was going to be a new DTrace capability, so future versions of > >hotuser/hotkernel should run the new DTrace feature if available. > > > >I originally wrote these tools a long time ago, and was never satisified > >with Perl post processing (I''m sure many of you have thought of > >writing these tools as well). Nor did I find a suitable DTrace hack > >to make it work efficiently from pure DTrace. Since the output is rather > >useful, I''ll bite the bullet and go with Perl post-processing for now... > > > >... > > > >enjoy! > > > >Brendan > > > > > > > >_______________________________________________ > >dtrace-discuss mailing list > >dtrace-discuss at opensolaris.org > > > > > >
Brendan Gregg wrote:>Using shell metacharacters within a dtrace -c won''t work, as dtrace >doesn''t invoke a sub shell. There is a way to redirect the dtrace probe >output to a different file descriptor: eg -o /dev/fd/2. > >That makes sense. Thanks Brendan. Chip