Hi folks- For a while, I have been trying to work out a method for observing suspend and resume failures. This has effectively been like pulling the monitor cable and storage devices, and trying to figure out what is happening up to a power failure. Up to this point, we have been using DEBUG printf''s to the serial port as it will work till the power goes out, and is easy to reprogram on restart. However, this limits usage to debug kernels, and doesn''t adapt very well to other possible devices. This morning I had the thought that maybe some or all of this functionality could be handled by having a ''logger'' provider in dtrace. My needs are to have a lightweight non-volatile logging mechansim that will continue to function while normal hardware I/O channels are being stopped or powered off. But I considered that this "feature" might be usefull to other consumers. The thought here is that this provider would have mechanisms for alternate places where dtrace data can be sent. In the case of s/r, the script could set the logger to be an onboard serial port, and dtrace output would go there (allowing me or other s/r users to actually use dtrace for debugging - including on production kernels). However, other loggers, such as USB, storage, or even network could be defined to suit a users needs. This is currently little more than just an idea without much thought as to how it might be implemented. However, I am interested in what the DTrace community thinks, including if there already is a mechanism that might do the same (remember, s/r needs are that this continues to work when most kernel services are off, and needs to be easily started from power-on). If this is a compelling feature, I would be happy to give it more thought (as well as solicitation on how to actually implement it). Cheers! ---- Randy
On Wed, Jun 10, 2009 at 04:15:30PM -0700, Randy Fishel wrote:> This morning I had the thought that maybe some or all of this > functionality could be handled by having a ''logger'' provider in > dtrace. My needs are to have a lightweight non-volatile logging > mechansim that will continue to function while normal hardware I/O > channels are being stopped or powered off. But I considered that this > "feature" might be usefull to other consumers.Have you looked at anonymous DTrace scripts? Those are used for tracing early in boot, and their results end up in a kernel-land buffer until retrieved from user-land once the system is up. You could use the same approach. One of the nice things about this is that if there''s a panic you can always get at the buffered data using kmdb (if you can''t use kmdb but can force a dump then you can always look at the data in the dump). Nico --
On Wed, 10 Jun 2009, Nicolas Williams wrote:> On Wed, Jun 10, 2009 at 04:15:30PM -0700, Randy Fishel wrote: > > This morning I had the thought that maybe some or all of this > > functionality could be handled by having a ''logger'' provider in > > dtrace. My needs are to have a lightweight non-volatile logging > > mechansim that will continue to function while normal hardware I/O > > channels are being stopped or powered off. But I considered that this > > "feature" might be usefull to other consumers. > > Have you looked at anonymous DTrace scripts? Those are used for tracing > early in boot, and their results end up in a kernel-land buffer until > retrieved from user-land once the system is up. You could use the same > approach. One of the nice things about this is that if there''s a panic > you can always get at the buffered data using kmdb (if you can''t use > kmdb but can force a dump then you can always look at the data in the > dump). > > Nico > -- >Anonymous DTrace is useful for early in the kernel startup, but what we have for suspend and resume is actually a running instance that is frozen. And in the midst of the freeze, the instance can be completely tossed and there is no core dump. A common example, is that we stop all hardware (turn off the monitor, and spin down disks), and then nothing. Power doesn''t go off and it looks like a hang. The only resort is to power-cycle, so there will be no core file, nor will there be any buffers for dtrace or kmdb to look at (same for resume, but that has the added burden of making sure the "logging" device gets restored to a usable state). The debug prototype continues to log messages to the serial port, so at least there is some idea as to what it was last doing (granted, if the machine goes off to the BIOS, all hope really is lost). But this is only useful in debug kernels, and there are many failures in the field that I haven''t a clue how to solve, as I don''t know where it gets lost. So even though some form of anonymous tracing will be used (it will need to be all in the kernel, as the first thing we stop is userland), without a non-volatile logging mechanism, it is a core that goes into the ether. Where I eventually landed, is that I need to have (or implement) a mechanism to send dtrace information to an arbitrarily defined streaming device. I was considering it as part of a ''power management'' provider, but it seemed this might be useful outside of a pm provider. Cheers! ---- Randy
Do you mean that the raw data would be sent (rather than being placed into buffers?) or that the dtrace util would spit out it''s info to the port? If the former, you will be limiting the acquisition of data to the baud rate/buffering capability of the serial driver, no? On Wed, Jun 10, 2009 at 8:09 PM, Randy Fishel <randy.fishel at sun.com> wrote:> > > On Wed, 10 Jun 2009, Nicolas Williams wrote: > > > On Wed, Jun 10, 2009 at 04:15:30PM -0700, Randy Fishel wrote: > > > This morning I had the thought that maybe some or all of this > > > functionality could be handled by having a ''logger'' provider in > > > dtrace. My needs are to have a lightweight non-volatile logging > > > mechansim that will continue to function while normal hardware I/O > > > channels are being stopped or powered off. But I considered that this > > > "feature" might be usefull to other consumers. > > > > Have you looked at anonymous DTrace scripts? Those are used for tracing > > early in boot, and their results end up in a kernel-land buffer until > > retrieved from user-land once the system is up. You could use the same > > approach. One of the nice things about this is that if there''s a panic > > you can always get at the buffered data using kmdb (if you can''t use > > kmdb but can force a dump then you can always look at the data in the > > dump). > > > > Nico > > -- > > > > > Anonymous DTrace is useful for early in the kernel startup, but what > we have for suspend and resume is actually a running instance that is > frozen. And in the midst of the freeze, the instance can be > completely tossed and there is no core dump. A common example, is > that we stop all hardware (turn off the monitor, and spin down disks), > and then nothing. Power doesn''t go off and it looks like a hang. The > only resort is to power-cycle, so there will be no core file, nor will > there be any buffers for dtrace or kmdb to look at (same for resume, > but that has the added burden of making sure the "logging" device > gets restored to a usable state). > > The debug prototype continues to log messages to the serial port, so > at least there is some idea as to what it was last doing (granted, if > the machine goes off to the BIOS, all hope really is lost). But this > is only useful in debug kernels, and there are many failures in the > field that I haven''t a clue how to solve, as I don''t know where it > gets lost. > > So even though some form of anonymous tracing will be used (it will > need to be all in the kernel, as the first thing we stop is userland), > without a non-volatile logging mechanism, it is a core that goes into > the ether. > > > > Where I eventually landed, is that I need to have (or implement) a > mechanism to send dtrace information to an arbitrarily defined > streaming device. I was considering it as part of a ''power > management'' provider, but it seemed this might be useful outside of a > pm provider. > > > Cheers! > > > ---- Randy > > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org >-- colin.burgess at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20090611/9254deca/attachment.html>
On Wed, Jun 10, 2009 at 05:09:38PM -0700, Randy Fishel wrote:> Where I eventually landed, is that I need to have (or implement) a > mechanism to send dtrace information to an arbitrarily defined > streaming device. I was considering it as part of a ''power > management'' provider, but it seemed this might be useful outside of a > pm provider.I agree. Simply provide some encoding of the trace data and write it wherever you can.
On Thu, Jun 11, 2009 at 01:15:14PM -0400, Colin Burgess wrote:> Do you mean that the raw data would be sent (rather than being placed into > buffers?) or that the dtrace util would spit out it''s info to the port?The user-land part of dtrace wouldn''t be running, so I''m sure Randy meant the former.> If the former, you will be limiting the acquisition of data to the baud > rate/buffering capability of the serial driver, no?Wouldn''t that be true regardless?
In the kernel. I am executing in function foo. foo calls putnext. I want a predicate that allows my fbt::putnext:entry to fire only when the parent is foo. Any suggestions ? I know about caller, but it is a PC. How do I tie that to foo ? rick -- Rickey C. Weisner Software Development and Performance Specialist Principal Field Technologist Systems Quality Office cell phone: 615-308-1147 email: rick.weisner at sun.com
would
fbt::foo:entry
{
self->infoo=1;
}
fbt::foo:return
/self->infoo/
{
self->infoo=0;
}
fbt::putnext:entry
/self->infoo/
{
do what you need
}
do the trick for you?
-Angelo
On Jun 12, 2009, at 3:44 PM, rickey c weisner wrote:
> In the kernel.
> I am executing in function foo.
>
> foo calls putnext.
>
> I want a predicate that allows my
> fbt::putnext:entry
> to fire only when the parent is foo.
>
> Any suggestions ?
>
> I know about caller, but it is a PC. How do I tie that to foo ?
>
> rick
>
> --
>
> Rickey C. Weisner
> Software Development and Performance Specialist
> Principal Field Technologist
> Systems Quality Office
> cell phone: 615-308-1147
> email: rick.weisner at sun.com
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
rickey c weisner writes:> In the kernel. > I am executing in function foo. > > foo calls putnext. > > I want a predicate that allows my > fbt::putnext:entry > to fire only when the parent is foo. > > Any suggestions ?fbt::foo:entry { self->flag++; } fbt::putnext:entry /self->flag/ { printf("hi there"); } fbt::foo:return { self->flag--; } -- James Carlson, Solaris Networking <james.d.carlson at sun.com> Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677
Thanks but I am already in foo so foo:entry does not fire. rick> > fbt::foo:entry > { > self->infoo=1; > } > > fbt::foo:return > /self->infoo/ > { > self->infoo=0; > } > > fbt::putnext:entry > /self->infoo/ > { > do what you need > } > > do the trick for you? > > -Angelo > > On Jun 12, 2009, at 3:44 PM, rickey c weisner wrote: > >> In the kernel. >> I am executing in function foo. >> >> foo calls putnext. >> >> I want a predicate that allows my >> fbt::putnext:entry >> to fire only when the parent is foo. >> >> Any suggestions ? >> >> I know about caller, but it is a PC. How do I tie that to foo ? >> >> rick >> >> -- >> >> Rickey C. Weisner >> Software Development and Performance Specialist >> Principal Field Technologist >> Systems Quality Office >> cell phone: 615-308-1147 >> email: rick.weisner at sun.com >> _______________________________________________ >> dtrace-discuss mailing list >> dtrace-discuss at opensolaris.org >-- Rickey C. Weisner Software Development and Performance Specialist Principal Field Technologist Systems Quality Office cell phone: 615-308-1147 email: rick.weisner at sun.com
On Fri, Jun 12, 2009 at 03:48:42PM -0400, James Carlson wrote:> rickey c weisner writes: > > In the kernel. > > I am executing in function foo. > > > > foo calls putnext. > > > > I want a predicate that allows my > > fbt::putnext:entry > > to fire only when the parent is foo. > > > > Any suggestions ? > > fbt::foo:entry > { > self->flag++; > } > > fbt::putnext:entry > /self->flag/ > { > printf("hi there"); > } > > fbt::foo:returnAlways predicate this with ''/self->flag/'', or threads which are currently in ''foo'' when tracing is started will trace all activity after it returns.> { > self->flag--; > }Cheers, - jonathan
Jonathan Adams writes:> Always predicate this with ''/self->flag/'', or threads which are currently in > ''foo'' when tracing is started will trace all activity after it returns.Yep; noted by meem as well. Sorry about that omission. -- James Carlson, Solaris Networking <james.d.carlson at sun.com> Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677
Let me try this again. I want a predicate such that if my probe is fbt::foo:entry the predicate will only fire if the parent of foo is foobar. Now I am already in foobar, I have been there for minutes. I want to see what is happenning underneath foobar. I can not set a flag when I enter foobar BECAUSE I am already running inside of foobar. The only thing I can thing of is to identify the threadid of interest and then key on the threadid. I had hoped for some magic with caller, but it is a hex address not a string. I guess I could determine the hex address range of foobar and key on that. Thanks, rick -- Rickey C. Weisner Software Development and Performance Specialist Principal Field Technologist Systems Quality Office cell phone: 615-308-1147 email: rick.weisner at sun.com
rickey c weisner wrote:> Let me try this again. > > I want a predicate such that if my probe is > fbt::foo:entry > the predicate will only fire if the parent of foo is foobar. > > Now I am already in foobar, I have been there for minutes. > I want to see what is happenning underneath foobar. > > I can not set a flag when I enter foobar BECAUSE I am already > running inside of foobar. > > The only thing I can thing of is to identify the threadid of > interest and then key on the threadid. > > I had hoped for some magic with caller, but it is a hex address > not a string. > > I guess I could determine the hex address range of foobar and > key on that. > > Thanks, > rickIt''s a bit difficult without seeing the actual code, but you could try something like this: fbt::foobar:entry { self->t = 5; } fbt:::entry /self->t/ { self->t--; printf("%s\n", probefunc); } 5 being the depth of the ''stack trace''. Rafael
rickey c weisner wrote:> Let me try this again. > > I want a predicate such that if my probe is > fbt::foo:entry > the predicate will only fire if the parent of foo is foobar. > > Now I am already in foobar, I have been there for minutes. > I want to see what is happenning underneath foobar. > > I can not set a flag when I enter foobar BECAUSE I am already > running inside of foobar. > > The only thing I can thing of is to identify the threadid of > interest and then key on the threadid. > > I had hoped for some magic with caller, but it is a hex address > not a string. > > I guess I could determine the hex address range of foobar and > key on that. > > Thanks, > rick >It seems like a tid would work out ok, especially if -- as you say -- foobar has been running for a long time. Just fire up dtrace twice: the first time, print backtrace and tid the first time any thread calls foo. Pick out the tid of interest and pass that as the $1 of a second .d which actually does what you want. If foo doesn''t have too many other callers, you could also instrument all of them and then have fbt::foo:entry/!self->caller_to_ignore/ { ... } I could imagine other variations on that theme, but at that point it would matter a lot who ''foo'' and ''foobar'' are and how they behave. Regards, Ryan Ryan
On Tue, Jun 16, 2009 at 09:46:13AM -0500, rickey c weisner wrote:> Let me try this again. > > I want a predicate such that if my probe is > fbt::foo:entry > the predicate will only fire if the parent of foo is foobar. > > Now I am already in foobar, I have been there for minutes. > I want to see what is happenning underneath foobar. > > I can not set a flag when I enter foobar BECAUSE I am already > running inside of foobar. > > The only thing I can thing of is to identify the threadid of > interest and then key on the threadid. > > I had hoped for some magic with caller, but it is a hex address > not a string. > > I guess I could determine the hex address range of foobar and > key on that.# dtrace -n'' fbt::kmem_alloc:return /((uintptr_t)caller - (uintptr_t)&`poll_common) < sizeof (`poll_common)/ {stack()} '' (replace kmem_alloc and poll_common with the functions of interest, of course) Cheers, - jonathan
Jonathan Adams wrote:> # dtrace -n'' > fbt::kmem_alloc:return > /((uintptr_t)caller - (uintptr_t)&`poll_common) < sizeof (`poll_common)/ > {stack()} > '' > > (replace kmem_alloc and poll_common with the functions of interest, of course) > Cheers, > - jonathanAh, very cool. Didn''t know about caller. Out of curiosity, why are you casting it ? Thanks, Rafael
On Tue, Jun 16, 2009 at 01:32:29PM -0300, Rafael Vanoni wrote:> Jonathan Adams wrote: > ># dtrace -n'' > > fbt::kmem_alloc:return > > /((uintptr_t)caller - (uintptr_t)&`poll_common) < sizeof > > (`poll_common)/ > > {stack()} > >'' > > > >(replace kmem_alloc and poll_common with the functions of interest, of > >course) > >Cheers, > >- jonathan > > Ah, very cool. Didn''t know about caller. > > Out of curiosity, why are you casting it ?Because you can''t subtract a pointer from an integer, or two incompatible pointer types? Cheers, - jonathan
Jonathan Adams wrote:> On Tue, Jun 16, 2009 at 01:32:29PM -0300, Rafael Vanoni wrote: >> Jonathan Adams wrote: >>> # dtrace -n'' >>> fbt::kmem_alloc:return >>> /((uintptr_t)caller - (uintptr_t)&`poll_common) < sizeof >>> (`poll_common)/ >>> {stack()} >>> '' >>> >>> (replace kmem_alloc and poll_common with the functions of interest, of >>> course) >>> Cheers, >>> - jonathan >> Ah, very cool. Didn''t know about caller. >> >> Out of curiosity, why are you casting it ? > > Because you can''t subtract a pointer from an integer, or two incompatible > pointer types? > > Cheers, > - jonathan >Yes, but isn''t caller already declared as uintptr_t ? Raf
On Tue, Jun 16, 2009 at 03:10:22PM -0300, Rafael Vanoni wrote:> Jonathan Adams wrote: > >On Tue, Jun 16, 2009 at 01:32:29PM -0300, Rafael Vanoni wrote: > >>Jonathan Adams wrote: > >>># dtrace -n'' > >>> fbt::kmem_alloc:return > >>> /((uintptr_t)caller - (uintptr_t)&`poll_common) < sizeof > >>> (`poll_common)/ > >>> {stack()} > >>>'' > >>> > >>>(replace kmem_alloc and poll_common with the functions of interest, of > >>>course) > >>>Cheers, > >>>- jonathan > >>Ah, very cool. Didn''t know about caller. > >> > >>Out of curiosity, why are you casting it ? > > > >Because you can''t subtract a pointer from an integer, or two incompatible > >pointer types? > > > >Cheers, > >- jonathan > > > > Yes, but isn''t caller already declared as uintptr_t ?Oh. Probably. caller() is a caddr_t in the kernel, hence my confusion. Cheers, - jonathan