Hi all,
IHAC whose machines are rebooting several times a day without anything useful
in the messages or on the console. They *suspect* it''s a 3rd party
product,
but cannot prove it and that OEM is pointing our way ...
What we see on the console is:
Sep 15 22:06:06 <host> last message repeated 2 times
Sep 15 22:06:06 <host> root: Running CRSD with TZ = Europe/Vienna
rebooting...
Resetting ...
not really helpful ...
Since cust is prepared to have the machine panic when it reboots, I tried
Dtrace on
fbt::kadmin:entry/arg0 < 10/{panic()}
("arg0 < 10" to avoid A_SWAPCTL panicing the machine), but even
making this
anonymous and following Jonathan Adams'' guide on reloading DTrace
(which
worked - I saw messages to that effect) didn''t trigger a panic when I
ran
"reboot" (as a test).
Can anyone supply a hint on how to further debug this?
TIA
Michael
--
Michael Schuster
Recursion, n.: see ''Recursion''
Hi,
I can use the D-program below to trace all USER
modules associated with any executable command
such as "ls":
#!/usr/sbin/dtrace -s
#pragma D option quiet
syscall::resolvepath:entry
/!progenyof($pid)/
{
self->umod = arg0;
}
syscall::resolvepath:return
/!progenyof($pid) && self->umod && arg1 >= 0/
{
printf("%s: %s\n", execname, copyinstr(self->umod));
self->umod = 0;
}
My question is:
How to trace all KERNEL modules associated with
any executable command?
Any hint will be greatly appreciated?
Thanks.
************************************************
* C P Lin, Common Technology Project Lead. *
* Sun Microsystems Inc. *
* E-Mail: c.lin at sun.com *
* Address: 4150 Network Circle, M/S UMPK12-330 *
* Santa Clara, CA 95054 *
* Phone: 650/352-4967 Fax: 650/786-7816 *
************************************************
You''re interested in seeing what kernel modules are involved with the
execution of the program? You could do something like this:
fbt:::entry
{
@[execname, probemod] = count();
}
This is a fairly big hammer since it instruments every function in the kernel;
what problem are you trying to solve exactly?
Adam
On Mon, Sep 18, 2006 at 09:47:19AM -0700, CHIAN-PHON LIN
wrote:> Hi,
>
> I can use the D-program below to trace all USER
> modules associated with any executable command
> such as "ls":
>
> #!/usr/sbin/dtrace -s
> #pragma D option quiet
> syscall::resolvepath:entry
> /!progenyof($pid)/
> {
> self->umod = arg0;
> }
> syscall::resolvepath:return
> /!progenyof($pid) && self->umod && arg1 >= 0/
> {
> printf("%s: %s\n", execname, copyinstr(self->umod));
> self->umod = 0;
> }
>
> My question is:
>
> How to trace all KERNEL modules associated with
> any executable command?
>
> Any hint will be greatly appreciated?
>
> Thanks.
>
> ************************************************
> * C P Lin, Common Technology Project Lead. *
> * Sun Microsystems Inc. *
> * E-Mail: c.lin at sun.com *
> * Address: 4150 Network Circle, M/S UMPK12-330 *
> * Santa Clara, CA 95054 *
> * Phone: 650/352-4967 Fax: 650/786-7816 *
> ************************************************
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
--
Adam Leventhal, Solaris Kernel Development http://blogs.sun.com/ahl
Adam, Please see my in-line comments: Adam Leventhal wrote On 09/19/06 11:15,:> You''re interested in seeing what kernel modules are involved with the > execution of the program? You could do something like this: > > fbt:::entry > { > @[execname, probemod] = count(); > } > > This is a fairly big hammer since it instruments every function in the kernel;I agreed and this may not be desirable for my application.> what problem are you trying to solve exactly?I wrote Qtrace (a test coverage tool) using Dtrace as one of the major underline supports. The first thing user want to know is the list of user and kernel modules available in a system. We can statically list all available fbt modules in the system using "dtrace -l" and dynamically scan all touched user modules during a period of time using syscall::resolvepath:entry with little performance panelty. However, we currently do not have a good way to list all touched kernel modules during a period of time and what you suggested may introduce too much system overhead. One solution I can think of is to filter out unix and genunix in fbt:???::entry statement to reduce most of the overhead since unix and genunix are going to be touched anyway. Is there a better way to do this? or Is it possible to achieve this without using Dtrace? Thanks.> Adam > > On Mon, Sep 18, 2006 at 09:47:19AM -0700, CHIAN-PHON LIN wrote: > >>Hi, >> >>I can use the D-program below to trace all USER >>modules associated with any executable command >>such as "ls": >> >>#!/usr/sbin/dtrace -s >>#pragma D option quiet >>syscall::resolvepath:entry >>/!progenyof($pid)/ >>{ >> self->umod = arg0; >>} >>syscall::resolvepath:return >>/!progenyof($pid) && self->umod && arg1 >= 0/ >>{ >> printf("%s: %s\n", execname, copyinstr(self->umod)); >> self->umod = 0; >>} >> >>My question is: >> >>How to trace all KERNEL modules associated with >>any executable command? >> >>Any hint will be greatly appreciated? >> >>Thanks. >> >>************************************************ >>* C P Lin, Common Technology Project Lead. * >>* Sun Microsystems Inc. * >>* E-Mail: c.lin at sun.com * >>* Address: 4150 Network Circle, M/S UMPK12-330 * >>* Santa Clara, CA 95054 * >>* Phone: 650/352-4967 Fax: 650/786-7816 * >>************************************************ >>_______________________________________________ >>dtrace-discuss mailing list >>dtrace-discuss at opensolaris.org > >-- ************************************************ * C P Lin, Common Technology Project Lead. * * Sun Microsystems Inc. * * E-Mail: c.lin at sun.com * * Address: 4150 Network Circle, M/S UMPK12-330 * * Santa Clara, CA 95054 * * Phone: 650/352-4967 Fax: 650/786-7816 * ************************************************
CHIAN-PHON LIN wrote:>> what problem are you trying to solve exactly? > > I wrote Qtrace (a test coverage tool) using Dtrace as one of > the major underline supports. The first thing user want to > know is the list of user and kernel modules available in a > system. We can statically list all available fbt modules > in the system using "dtrace -l" and dynamically scan all > touched user modules during a period of time using > syscall::resolvepath:entry with little performance panelty. > However, we currently do not have a good way to list all > touched kernel modules during a period of timeI may be especially thick this morning: what exactly do you mean by "kernel modules touched"? Micahel -- Michael Schuster Recursion, n.: see ''Recursion''
Michael, Michael Schuster wrote On 09/19/06 22:48,:> CHIAN-PHON LIN wrote: > >>> what problem are you trying to solve exactly? >> >> >> I wrote Qtrace (a test coverage tool) using Dtrace as one of >> the major underline supports. The first thing user want to >> know is the list of user and kernel modules available in a >> system. We can statically list all available fbt modules >> in the system using "dtrace -l" and dynamically scan all >> touched user modules during a period of time using >> syscall::resolvepath:entry with little performance panelty. >> However, we currently do not have a good way to list all >> touched kernel modules during a period of time > > > I may be especially thick this morning: what exactly do you mean by > "kernel modules touched"?"Kernel module" is the module which belongs to the OS such as unix or device driver. If any function within a kernel module is called then we say that module is touched or traced. Dtrace allows us to know which module is touched. Thanks.> Micahel-- ************************************************ * C P Lin, Common Technology Project Lead. * * Sun Microsystems Inc. * * E-Mail: c.lin at sun.com * * Address: 4150 Network Circle, M/S UMPK12-330 * * Santa Clara, CA 95054 * * Phone: 650/352-4967 Fax: 650/786-7816 * ************************************************
Adam, Adam Leventhal wrote On 09/19/06 11:15,:> You''re interested in seeing what kernel modules are involved with the > execution of the program? You could do something like this: > > fbt:::entry > { > @[execname, probemod] = count(); > } > > This is a fairly big hammer since it instruments every function in the kernel;By simply excluding genunix and unix from module field drastically reduced overhead as below: #!/usr/sbin/dtrace -Zs #pragma D option quiet fbt:*[!x]::entry, fbt:*[!i]x::entry, fbt:*[!n]ix::entry, fbt:*[!u]nix::entry /!progenyof($pid)/ { @[execname, probemod] = count(); } Thanks. ************************************************ * C P Lin, Common Technology Project Lead. * * Sun Microsystems Inc. * * E-Mail: c.lin at sun.com * * Address: 4150 Network Circle, M/S UMPK12-330 * * Santa Clara, CA 95054 * * Phone: 650/352-4967 Fax: 650/786-7816 * ************************************************
Don''t you need to limit to kernel threads associated with your target
process?
#!/usr/sbin/dtrace -Zs
#pragma D option quiet
fbt:*[!x]::entry,
fbt:*[!i]x::entry,
fbt:*[!n]ix::entry,
fbt:*[!u]nix::entry
/ $target == pid /
{
@[execname, probemod] = count();
}
then invoke with the "-c progname" option.
Chip
CHIAN-PHON LIN wrote:> Adam,
>
> Adam Leventhal wrote On 09/19/06 11:15,:
>> You''re interested in seeing what kernel modules are involved
with the
>> execution of the program? You could do something like this:
>>
>> fbt:::entry
>> {
>> @[execname, probemod] = count();
>> }
>>
>> This is a fairly big hammer since it instruments every function in
>> the kernel;
>
> By simply excluding genunix and unix from module field
> drastically reduced overhead as below:
>
> #!/usr/sbin/dtrace -Zs
> #pragma D option quiet
> fbt:*[!x]::entry,
> fbt:*[!i]x::entry,
> fbt:*[!n]ix::entry,
> fbt:*[!u]nix::entry
> /!progenyof($pid)/
> {
> @[execname, probemod] = count();
> }
>
> Thanks.
>
> ************************************************
> * C P Lin, Common Technology Project Lead. *
> * Sun Microsystems Inc. *
> * E-Mail: c.lin at sun.com *
> * Address: 4150 Network Circle, M/S UMPK12-330 *
> * Santa Clara, CA 95054 *
> * Phone: 650/352-4967 Fax: 650/786-7816 *
> ************************************************
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
Hi Chip, The point of including execname as a key to the aggregation is because we apparently want to see what kernel modules are being used by all processes except for those spawned by dtrace(1m) itself -- how this information might then be used is a bit a mystery to me. Adam On Wed, Sep 20, 2006 at 11:30:14AM -0500, Chip Bennett wrote:> Don''t you need to limit to kernel threads associated with your target > process? > > #!/usr/sbin/dtrace -Zs > #pragma D option quiet > fbt:*[!x]::entry, > fbt:*[!i]x::entry, > fbt:*[!n]ix::entry, > fbt:*[!u]nix::entry > / $target == pid / > { > @[execname, probemod] = count(); > } > > then invoke with the "-c progname" option. > > Chip > > > CHIAN-PHON LIN wrote: > >Adam, > > > >Adam Leventhal wrote On 09/19/06 11:15,: > >>You''re interested in seeing what kernel modules are involved with the > >>execution of the program? You could do something like this: > >> > >>fbt:::entry > >>{ > >> @[execname, probemod] = count(); > >>} > >> > >>This is a fairly big hammer since it instruments every function in > >>the kernel; > > > >By simply excluding genunix and unix from module field > >drastically reduced overhead as below: > > > >#!/usr/sbin/dtrace -Zs > >#pragma D option quiet > >fbt:*[!x]::entry, > >fbt:*[!i]x::entry, > >fbt:*[!n]ix::entry, > >fbt:*[!u]nix::entry > >/!progenyof($pid)/ > >{ > > @[execname, probemod] = count(); > >} > > > >Thanks. > > > >************************************************ > >* C P Lin, Common Technology Project Lead. * > >* Sun Microsystems Inc. * > >* E-Mail: c.lin at sun.com * > >* Address: 4150 Network Circle, M/S UMPK12-330 * > >* Santa Clara, CA 95054 * > >* Phone: 650/352-4967 Fax: 650/786-7816 * > >************************************************ > >_______________________________________________ > >dtrace-discuss mailing list > >dtrace-discuss at opensolaris.org > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org-- Adam Leventhal, Solaris Kernel Development http://blogs.sun.com/ahl
Adam, I hadn''t noticed that execname was one of aggregation keys. That would have been a clue that the goal was something different than what I thought. Thanks, Chip Adam Leventhal wrote:> Hi Chip, > > The point of including execname as a key to the aggregation is because > we apparently want to see what kernel modules are being used by all > processes except for those spawned by dtrace(1m) itself -- how this > information might then be used is a bit a mystery to me. > > Adam > > On Wed, Sep 20, 2006 at 11:30:14AM -0500, Chip Bennett wrote: > >> Don''t you need to limit to kernel threads associated with your target >> process? >> >> #!/usr/sbin/dtrace -Zs >> #pragma D option quiet >> fbt:*[!x]::entry, >> fbt:*[!i]x::entry, >> fbt:*[!n]ix::entry, >> fbt:*[!u]nix::entry >> / $target == pid / >> { >> @[execname, probemod] = count(); >> } >> >> then invoke with the "-c progname" option. >> >> Chip >> >> >> CHIAN-PHON LIN wrote: >> >>> Adam, >>> >>> Adam Leventhal wrote On 09/19/06 11:15,: >>> >>>> You''re interested in seeing what kernel modules are involved with the >>>> execution of the program? You could do something like this: >>>> >>>> fbt:::entry >>>> { >>>> @[execname, probemod] = count(); >>>> } >>>> >>>> This is a fairly big hammer since it instruments every function in >>>> the kernel; >>>> >>> By simply excluding genunix and unix from module field >>> drastically reduced overhead as below: >>> >>> #!/usr/sbin/dtrace -Zs >>> #pragma D option quiet >>> fbt:*[!x]::entry, >>> fbt:*[!i]x::entry, >>> fbt:*[!n]ix::entry, >>> fbt:*[!u]nix::entry >>> /!progenyof($pid)/ >>> { >>> @[execname, probemod] = count(); >>> } >>> >>> Thanks. >>> >>> ************************************************ >>> * C P Lin, Common Technology Project Lead. * >>> * Sun Microsystems Inc. * >>> * E-Mail: c.lin at sun.com * >>> * Address: 4150 Network Circle, M/S UMPK12-330 * >>> * Santa Clara, CA 95054 * >>> * Phone: 650/352-4967 Fax: 650/786-7816 * >>> ************************************************ >>> _______________________________________________ >>> dtrace-discuss mailing list >>> dtrace-discuss at opensolaris.org >>> >> _______________________________________________ >> dtrace-discuss mailing list >> dtrace-discuss at opensolaris.org >> > >
Hi Michael, You could try putting using the fbt::mdboot:entry probe instead -- it might be worth a shot. Adam On Mon, Sep 18, 2006 at 04:42:28PM +0200, Michael Schuster wrote:> Hi all, > > IHAC whose machines are rebooting several times a day without anything > useful in the messages or on the console. They *suspect* it''s a 3rd party > product, but cannot prove it and that OEM is pointing our way ... > > What we see on the console is: > > Sep 15 22:06:06 <host> last message repeated 2 times > Sep 15 22:06:06 <host> root: Running CRSD with TZ = Europe/Vienna > rebooting... > Resetting ... > > not really helpful ... > > Since cust is prepared to have the machine panic when it reboots, I tried > Dtrace on > > fbt::kadmin:entry/arg0 < 10/{panic()} > > ("arg0 < 10" to avoid A_SWAPCTL panicing the machine), but even making this > anonymous and following Jonathan Adams'' guide on reloading DTrace (which > worked - I saw messages to that effect) didn''t trigger a panic when I ran > "reboot" (as a test). > > Can anyone supply a hint on how to further debug this? > > TIA > Michael > -- > Michael Schuster > Recursion, n.: see ''Recursion'' > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org-- Adam Leventhal, Solaris Kernel Development http://blogs.sun.com/ahl
Adam Leventhal wrote:> Hi Michael, > > You could try putting using the fbt::mdboot:entry probe instead -- it > might be worth a shot.it was - but I "implemented" it with kmdb :-) - in my tests, I got results quicker that way; it may be just me, but getting DTrace to actually produce anything useful at that late a point in a system''s life proved rather painful, and I think I wouldn''t have got anything without a dump anyway, even with DTrace (pls. contradict me if I''m wrong, and show me how ;-), so instead of #!/usr/bin/dtrace -Aws fbt::mdboot:entry, fbt::zone_shutdown_global:entry { printf("%s", execname); panic(); } (I added zone_shutdown_global to catch as many processes as possible if a regular "init 6" had been done) and the rather tiresome sequence to get dtrace reload without booting (which I didn''t manage successfully every time), I told cust to # mdb -K Welcome to kmdb [1]> mdboot::bp -d -c ''$<systemdump'' [1]> zone_shutdown_global::bp -d -c ''$<systemdump'' [1]> [1]> :c which gave us a nice crash dump, and showed that Oracle''s "oprocd" is causing this. We only got these results yesterday, so I hadn''t gotten round to posting an update yet. thx Michael -- Michael Schuster +49 89 46008-2974 / x62974 visit the online support center: http://www.sun.com/osc/ Recursion, n.: see ''Recursion''