I''ve been looking at the anonymous tracing section about using dtrace during boot. From what I can tell, is that after you enable it, reboot the host, you can run dtrace after you get a command prompt. Unfortunately in my situation, the system hard hangs during the boot phase, specifically, during configuration of devices. Is there a way to instrument dtrace during the boot phase to print out, say fbt of what devfsadm is doing? I would like to see what device it may be hanging on and what function? Thanks Dave
Unfortunately I can''t send a break and force a crash dump. Chip Bennett wrote:> I wonder if you could combine anonymous tracing with postmortem tracing > to get the desired result. > > Use anonymous tracing to collect the anonymous trace buffer, and then > when the system hangs, generate a crash dump and use the mdb command to > display the buffer, as described in the postmortem technique. > > Chip > > David Blasingame wrote: > >> I''ve been looking at the anonymous tracing section about using dtrace >> during boot. From what I can tell, is that after you enable it, >> reboot the host, you can run dtrace after you get a command prompt. >> >> Unfortunately in my situation, the system hard hangs during the boot >> phase, specifically, during configuration of devices. Is there a way >> to instrument dtrace during the boot phase to print out, say fbt of >> what devfsadm is doing? I would like to see what device it may be >> hanging on and what function? >> >> Thanks >> >> Dave >> _______________________________________________ >> dtrace-discuss mailing list >> dtrace-discuss at opensolaris.org > >
I wonder if you could combine anonymous tracing with postmortem tracing to get the desired result. Use anonymous tracing to collect the anonymous trace buffer, and then when the system hangs, generate a crash dump and use the mdb command to display the buffer, as described in the postmortem technique. Chip David Blasingame wrote:> I''ve been looking at the anonymous tracing section about using dtrace > during boot. From what I can tell, is that after you enable it, > reboot the host, you can run dtrace after you get a command prompt. > > Unfortunately in my situation, the system hard hangs during the boot > phase, specifically, during configuration of devices. Is there a way > to instrument dtrace during the boot phase to print out, say fbt of > what devfsadm is doing? I would like to see what device it may be > hanging on and what function? > > Thanks > > Dave > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org
On 4/16/07, David Blasingame <David.Blasingame at sun.com> wrote:> Unfortunately I can''t send a break and force a crash dump. >Perhaps the dtrace program keeps running and you could could add something like the following to your script... #pragma D option destructive tick-300sec { panic() } Mike -- Mike Gerdts http://mgerdts.blogspot.com/
Mike, There''s no dtrace command running with anonymous tracing. Just an anonymous buffer is being collected for future analysis. David needs some way to capture the buffer so that he can run mdb-dtrace macros to display it later. Chip Mike Gerdts wrote:> On 4/16/07, David Blasingame <David.Blasingame at sun.com> wrote: >> Unfortunately I can''t send a break and force a crash dump. >> > > Perhaps the dtrace program keeps running and you could could add > something like the following to your script... > > #pragma D option destructive > > tick-300sec > { > panic() > } > > Mike >
On Mon, Apr 16, 2007 at 02:25:57PM -0500, Chip Bennett wrote:> There''s no dtrace command running with anonymous tracing. Just an > anonymous buffer is being collected for future analysis. David needs > some way to capture the buffer so that he can run mdb-dtrace macros to > display it later.The probes are executed though, and any actions that don''t require a user-land consumer (as system() does) should execute, no? And panic() shouldn''t require a user-land consumer, I''d expect.> Chip > > Mike Gerdts wrote: > >On 4/16/07, David Blasingame <David.Blasingame at sun.com> wrote: > >>Unfortunately I can''t send a break and force a crash dump. > >> > > > >Perhaps the dtrace program keeps running and you could could add > >something like the following to your script... > > > >#pragma D option destructive > > > >tick-300sec > >{ > > panic() > >} > > > >Mike > > > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org
Nicolas Williams wrote:> On Mon, Apr 16, 2007 at 02:25:57PM -0500, Chip Bennett wrote: > >> There''s no dtrace command running with anonymous tracing. Just an >> anonymous buffer is being collected for future analysis. David needs >> some way to capture the buffer so that he can run mdb-dtrace macros to >> display it later. >> > > The probes are executed though, and any actions that don''t require a > user-land consumer (as system() does) should execute, no? And panic() > shouldn''t require a user-land consumer, I''d expect. > >I think you are correct. It should work. Chip
On Mon, Apr 16, 2007 at 01:57:44PM -0500, David Blasingame wrote:> Unfortunately I can''t send a break and force a crash dump. > > Chip Bennett wrote: > >I wonder if you could combine anonymous tracing with postmortem tracing > >to get the desired result. > > > >Use anonymous tracing to collect the anonymous trace buffer, and then > >when the system hangs, generate a crash dump and use the mdb command to > >display the buffer, as described in the postmortem technique.I recommend using a profile probe with a panic() or breakpoint() action, e.g.: tick-1sec /n++ > 600/ { panic(); } Will force a panic after the machine has been up for ten minutes. Then you can use ::dtrace_state/::dtrace to look at the state in the dump (though, frankly, you will likely find $<threadlist and/or ::cpuinfo to be more useful). - Bryan -------------------------------------------------------------------------- Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc
I''ve been trying to prove to myself that anonymous tracing automatically gets a ring buffer, but I haven''t been able to. Can anyone confirm that? Also, what size buffer by default, and can you effect the size with the -b option? Thanks, Chip Bryan Cantrill wrote:> On Mon, Apr 16, 2007 at 01:57:44PM -0500, David Blasingame wrote: > >> Unfortunately I can''t send a break and force a crash dump. >> >> Chip Bennett wrote: >> >>> I wonder if you could combine anonymous tracing with postmortem tracing >>> to get the desired result. >>> >>> Use anonymous tracing to collect the anonymous trace buffer, and then >>> when the system hangs, generate a crash dump and use the mdb command to >>> display the buffer, as described in the postmortem technique. >>> > > I recommend using a profile probe with a panic() or breakpoint() action, > e.g.: > > tick-1sec > /n++ > 600/ > { > panic(); > } > > Will force a panic after the machine has been up for ten minutes. Then > you can use ::dtrace_state/::dtrace to look at the state in the dump > (though, frankly, you will likely find $<threadlist and/or ::cpuinfo to > be more useful). > > - Bryan > > -------------------------------------------------------------------------- > Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc >
Great idea, I''ll have to give it a shot. Hopefully it will create the panic I need for the hard hang. Thanks Dave Bryan Cantrill wrote:> On Mon, Apr 16, 2007 at 01:57:44PM -0500, David Blasingame wrote: > >> Unfortunately I can''t send a break and force a crash dump. >> >> Chip Bennett wrote: >> >>> I wonder if you could combine anonymous tracing with postmortem tracing >>> to get the desired result. >>> >>> Use anonymous tracing to collect the anonymous trace buffer, and then >>> when the system hangs, generate a crash dump and use the mdb command to >>> display the buffer, as described in the postmortem technique. >>> > > I recommend using a profile probe with a panic() or breakpoint() action, > e.g.: > > tick-1sec > /n++ > 600/ > { > panic(); > } > > Will force a panic after the machine has been up for ten minutes. Then > you can use ::dtrace_state/::dtrace to look at the state in the dump > (though, frankly, you will likely find $<threadlist and/or ::cpuinfo to > be more useful). > > - Bryan > > -------------------------------------------------------------------------- > Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org >
Hi Chip,> I''ve been trying to prove to myself that anonymous tracing > automatically gets a ring buffer, but I haven''t been able to. Can > anyone confirm that? Also, what size buffer by default, and can you > effect the size with the -b option?No, I think it gets the default policy of "switch" and the buffers, by default, are 4M. I''ve got a prototype dcmd which lets you look at the value of options that are set: > ::dtrace_state ADDR MINOR PROC NAME FILE fffffffec2824400 2 - <anonymous> - > fffffffec2824400::dtrace_options bufsize 4194304 bufpolicy 2 dynvarsize 1048576 <chop> I''ve got to transform the values into a more human readable format (e.g. 4M instead of 4194304). A bufpolicy of 2 is "switch". Yes, you can tune the buffer size as you would normally. Cheers. Jon.> > Thanks, > Chip > > Bryan Cantrill wrote: >> On Mon, Apr 16, 2007 at 01:57:44PM -0500, David Blasingame wrote: >> >>> Unfortunately I can''t send a break and force a crash dump. >>> >>> Chip Bennett wrote: >>> >>>> I wonder if you could combine anonymous tracing with postmortem >>>> tracing to get the desired result. >>>> >>>> Use anonymous tracing to collect the anonymous trace buffer, and >>>> then when the system hangs, generate a crash dump and use the mdb >>>> command to display the buffer, as described in the postmortem >>>> technique. >>>> >> >> I recommend using a profile probe with a panic() or breakpoint() action, >> e.g.: >> >> tick-1sec >> /n++ > 600/ >> { >> panic(); >> } >> >> Will force a panic after the machine has been up for ten minutes. Then >> you can use ::dtrace_state/::dtrace to look at the state in the dump >> (though, frankly, you will likely find $<threadlist and/or ::cpuinfo to >> be more useful). >> >> - Bryan >> >> -------------------------------------------------------------------------- >> >> Bryan Cantrill, Solaris Kernel Development. >> http://blogs.sun.com/bmc >> > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org