dtrace discuss - Jan 2006 - dtrace breakpoint() output availability in kmdb session?

When using dtrace breakpoint(), how do you flush/access information
printed by dtrace printf calls so that it can be used when you get
to the kmdb breakpoint?

For the bug I am working on, there is a IO framework memory leak
associated with the processing of one devinfo node during a branch DR
on an E15K.  The debug strategy was to use dtrace fbt to trigger a
breakpoint on the specific invocation of the function that performs
the allocation that leaks (the "iosram" node), and then continue
debugging in kmdb.  The following gets me to the breakpoint prior to
the leaky allocation and prints out the dip

dtrace -w -n fbt::brevq_enqueue:entry''\
    /((struct dev_info*)args[1])->devi_node_name=="iosram"/ { \
    printf("dtrace->kmdb on %s %p\n", \
    stringof(((struct dev_info *)args[1])->devi_node_name), \
    args[1]); \
    breakpoint(); }''  \
    -c ''cfgadm -c unconfigure IO1''

but I find that when I get to my breakpoint the
dtrace printf information is not displayed until I '':c".

Is there a dcmd (or something) that allows display of buffered
dtrace output from kmdb?  The ''ecb'' seems intriguing, but I
don''t know how to use it.

-Chris



# dtrace -w -n <<<see above>>>
dtrace: description ''fbt::brevq_enqueue:entry'' matched 1 probe
dtrace: allowing destructive actions
dtrace: breakpoint action at probe fbt:genunix:brevq_enqueue:entry (ecb
3001bc06458)
kmdb: target stopped at:
kaif_enter:     ta        %icc, %g0 + 0x7d
[34]> $c
kaif_enter(2a100cef3ac, 1000000000000000, 30, 1828400, 0, 18a7400)
dtrace`dtrace_action_breakpoint+0x20c(29, 53, 42, 48, 0, 70649b38)
dtrace`dtrace_probe+0xacc(0, 0, 3002067ab80, 1887e08, 18886c8, 3005d5dc038)
i_log_devfs_branch_remove+0x10(2a100cefac0, 300062b9800, 0, 0, 0, 6)
mt_config_children+0x90(60000bd9280, 0, 300062b9800, ffffffff, 20000000, 8000000
)
unconfig_grand_children+0x2c(60000bd9280, 2a100cefac8, 1040010, ffffffff,
2a100cefac0, 0)
devi_unconfig_common+0x8c(300062bcc00, 2a100cefac8, 1040010, ffffffff,
2a100cefac0, ffffffff)
mt_config_thread+0x7c(600001dd780, 2a100e075d0, 600007c47e8, 187c5c0,
300062bcc00, 0)
thread_start+4(600001dd780, 0, 0, 0, 0, 0)
[34]> :c
Jan 15 23:35:46 xc4p11-b2 dr: OS unconfigure dr at 0:IO1::pci0
Jan 15 23:35:46 xc4p11-b2 dr: OS unconfigure dr at 0:IO1::pci1
Jan 15 23:35:46 xc4p11-b2 dr: OS unconfigure dr at 0:IO1::pci2
Jan 15 23:35:46 xc4p11-b2 dr: OS unconfigure dr at 0:IO1::pci3
dtrace: pid 10791 has exited
CPU     ID                    FUNCTION:NAME
 34   5400              brevq_enqueue:entry dtrace->kmdb on iosram
300062b9800

Note how "kmdb on iosram 300062b9800" not available until after
'':c''
This message posted from opensolaris.org

> When using dtrace breakpoint(), how do you flush/access information
> printed by dtrace printf calls so that it can be used when you get
> to the kmdb breakpoint?
> 
> For the bug I am working on, there is a IO framework memory leak
> associated with the processing of one devinfo node during a branch DR
> on an E15K.  The debug strategy was to use dtrace fbt to trigger a
> breakpoint on the specific invocation of the function that performs
> the allocation that leaks (the "iosram" node), and then continue
> debugging in kmdb.  The following gets me to the breakpoint prior to
> the leaky allocation and prints out the dip
> 
> dtrace -w -n fbt::brevq_enqueue:entry''\
>     /((struct dev_info*)args[1])->devi_node_name=="iosram"/ {
\
>     printf("dtrace->kmdb on %s %p\n", \
>     stringof(((struct dev_info *)args[1])->devi_node_name), \
>     args[1]); \
>     breakpoint(); }''  \
>     -c ''cfgadm -c unconfigure IO1''
> 
> but I find that when I get to my breakpoint the
> dtrace printf information is not displayed until I '':c".
> 
> Is there a dcmd (or something) that allows display of buffered
> dtrace output from kmdb?  The ''ecb'' seems intriguing, but
I
> don''t know how to use it.
> 
> -Chris
DTrace printf() happens in userland, not the kernel: the arguments are
recorded into your per-cpu buffer and then later formatted by dtrace(1M).
So there is no flush you can apply: your printf and breakpoint will happen
as part of the same enabling processing the kernel.

Given your example above, I would suggest either just breakpointing and
retrieving what you need from the stack (i.e. $c in kmdb and find the
argument yourself), or debugging the entire thing from DTrace (i.e.
don''t
bother with breakpoint and just use DTrace to capture stack() or whatever
you need to debug the memory leak at the end of your DTrace experiment).

-Mike
 
-- 
Mike Shapiro, Solaris Kernel Development. blogs.sun.com/mws/