Can fbt trace on inlined functions? Assuming not, is there any prospect of it? It doesn''t feel impossible, with suitable cooperation from the compiler - though I can see that getting at args is a big issue. However, even without args capability it would be useful observability. If need be, I''ll raise an RFE. - Jeremy This message posted from opensolaris.org
fbt probes is specifically for tracing kernel functions boundaries so it''s not what you want. I do not think it''s likely to automatically detect the location of since inline functions in general since they are "included" into the code unless the code is compiled with "-g". The pid provider, however, can trace every assembly instructions so that may be what you are looking for. You still need to locate where the inline function "starts" so your debugger will be helpful here. -- Just me, Wire ... On 12/21/05, John Hall <jh9solex at wizmail.org> wrote:> Can fbt trace on inlined functions? > > Assuming not, is there any prospect of it? It doesn''t feel impossible, with suitable cooperation from > the compiler - though I can see that getting at args is a big issue. However, even without args > capability it would be useful observability. > > If need be, I''ll raise an RFE. > > - Jeremy > This message posted from opensolaris.org > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org >
Wee Yeh Tan wrote:> fbt probes is specifically for tracing kernel functions boundaries so > it''s not what you want.I beg to differ. I''d like to be able to see the boundaries of the functions *as they are written in source code*. The kernel-ness is irrelevant.> > I do not think it''s likely to automatically detect the location of > since inline functions in general since they are "included" into the > code unless the code is compiled with "-g".I agree that''s (probably) the current position. I''m asking for an improvement.> > The pid provider, however, can trace every assembly instructions so > that may be what you are looking for. You still need to locate where > the inline function "starts" so your debugger will be helpful here.No, that''s a step too complex. Cheers, Jeremy
> Wee Yeh Tan wrote: > > fbt probes is specifically for tracing kernel functions boundaries so > > it''s not what you want. > > I beg to differ. I''d like to be able to see the boundaries of > the functions *as they are written in source code*. > > The kernel-ness is irrelevant. > > > > > I do not think it''s likely to automatically detect the location of > > since inline functions in general since they are "included" into the > > code unless the code is compiled with "-g". > > I agree that''s (probably) the current position. I''m asking for > an improvement. > > > > > The pid provider, however, can trace every assembly instructions so > > that may be what you are looking for. You still need to locate where > > the inline function "starts" so your debugger will be helpful here. > > No, that''s a step too complex. > > Cheers, > JeremyCertainly from a developer perspective it might be desirable to see and be able to more clearly instrument inline function entry points. That said, it does start to tread beyond the semantic notion of a "function boundary" as defined by FBT and into telling potential lies. For example, if a small bit of code that does a few loads or stores gets inlined, a very aggressive back- end optimizer may end up reordering the surrounding instructions in such a way that the text of the inline is actually intermingled with the text of the caller. At this point, what does it mean to "enter" that inline?>From a DTrace implementation perspective, the issue is of course that we can''tuse pure binary analysis to detect inlines, as we do today for fbt''s detection of function entry and return locations. The only way to do this is to consume compiler/debugger annotations such as those provided by DWARF, which then leads to the need to have: (a) a krtld DWARF consumer or DWARF->CTF support for inlines (b) kernel modules must not have such information stripped in order to get the corresponding instrumentation (c) the information in (a) needs to remain in kernel memory so that it can be consumed when DTrace loads, increasing overhead (d) the necessity of dealing with varying forms of (a) from different compilers with different properties and bugs, etc. It''s possible, but there are a lot of drawbacks and it poses a significant additional maintenance burden given the current state of compiler annotation, with somewhat unclear benefit given what I describe above about semantics. It might be worth explaining some of the key use cases you see as important here, as perhaps there are some subcases we can address (or might be addressed by using DTrace with the Analyzer) without having to deal with (a-d). -Mike -- Mike Shapiro, Solaris Kernel Development. blogs.sun.com/mws/
Jeremy Harris wrote:> Wee Yeh Tan wrote: > >> fbt probes is specifically for tracing kernel functions boundaries so >> it''s not what you want. > > > I beg to differ. I''d like to be able to see the boundaries of > the functions *as they are written in source code*. > > The kernel-ness is irrelevant. >If you''ve read the DTrace guide for chapter 20, the fbt provider, the first sentence says: --- This chapter describes the Function Boundary Tracing (FBT) provider, which provides probes associated with the entry to and return from most functions in the Solaris kernel. --- So fbt is for "entry to and return from most functions in the Solaris kernel" If you''re looking at userland, then the pid provider is what you need.>> >> I do not think it''s likely to automatically detect the location of >> since inline functions in general since they are "included" into the >> code unless the code is compiled with "-g". > > > I agree that''s (probably) the current position. I''m asking for > an improvement. >DTrace does not depend on any compiler or product. If you''ve written the code, you can add probes anywhere you like, but the best case is to use the pid provider as most of what you need is there. Once a function has been inlined, there''s no longer a call stack, so there''s no way for DTrace to even know there is a boundary, especially if other optimizations are present. Thanks, Brian
Michael Shapiro wrote:> it > does start to tread beyond the semantic notion of a "function boundary" as > defined by FBT and into telling potential lies. For example, if a small bit > of code that does a few loads or stores gets inlined, a very aggressive back- > end optimizer may end up reordering the surrounding instructions in such a > way that the text of the inline is actually intermingled with the text of > the caller. At this point, what does it mean to "enter" that inline?I agree, but I''m not after purity here, merely utility. Some definition like "The earliest machine instruction attributable by the compiler to the inlined routine, as the entry, and the latest as the end" would be workable. Agreed there are issues with instructions being promoted to before branches. The point about optimizer re-ordering actions is valid, and I''m assuming from the start the need for compiler cooperation even to the level of it offering, purely for the probe point, an out-of-line code segment which if executed from the probe point gathers the "arguments" into a well-known set of registers.> need to have: > > (a) a krtld DWARF consumer or DWARF->CTF support for inlinesyes> (b) kernel modules must not have such information stripped > in order to get the corresponding instrumentationyes - it could be stored as separate files but I don''t see the point> (c) the information in (a) needs to remain in kernel memory > so that it can be consumed when DTrace loads, increasing overheadThat needs to be an option, but isn''t a requirement. It could be loaded on demand just as kernel modules are loaded on demand, which would be usable so long as the load process didn''t change the behaviour under study. Hence the normal overhead would be limited to any extra patch-point instructions in the module text.> (d) the necessity of dealing with varying forms of (a) from > different compilers with different properties and bugs, etc.Yes; this is the worst aspect. Support from a limited set of compilers would be likely, perhaps only Sun''s own. Obviously gcc would be a nice-to-have, as users of it tend to be more enthusiastic in the use of inlines.> It might be worth explaining some of the key use cases you see as important > here, as perhaps there are some subcases we can address (or might be addressed > by using DTrace with the Analyzer) without having to deal with (a-d).The very basic "dtrace -F" output displaying all functions entered/exited under some one specified function - nothing complex, just "where did my code go?" I''d note that it would also support the use of cross-procedural optimization and automatic inlining. I realise that engineering is wary of using such within the kernel (or even using higher levels of optimisation) but perhaps better tools would help. I can also envisage an mdb ::dis annotating a section of code as "inlined <func>". There''s a logical extension here to the capability to set an mdb breakpoint on all instances of an inline in one go, though I can an indexing issue with that. Brian Kolaci said:> So fbt is for "entry to and return from most functions in the Solaris > kernel" > If you''re looking at userland, then the pid provider is what you need.Yes, I''m coming at this from a kernel perspective. But doesn''t the pid provider suffer the same lack: inlined functions are invisible, at least by name, to it?> Once a function has been inlined, there''s no longer a call stack, so > there''s no way for DTrace to even know there is a boundary, especially > if other optimizations are present.Yes, at present. I''d like there to be, which requires keeping track of information which was once known to the compiler. - Jeremy This message posted from opensolaris.org
> The point about optimizer re-ordering actions is valid, and I''m assuming > from the start the need for compiler cooperation even to the level of it > offering, purely for the probe point, an out-of-line code segment > which if executed from the probe point gathers the "arguments" into > a well-known set of registers. > > > > need to have: > > > > (a) a krtld DWARF consumer or DWARF->CTF support for inlines > > yesMike wasn''t actually checking with you, he was telling you that this is a hell of a lot of work.> > (b) kernel modules must not have such information stripped > > in order to get the corresponding instrumentation > > yes - it could be stored as separate files but I don''t see the pointAgain, the point is that this is a lot of work.> > It might be worth explaining some of the key use cases you see as important > > here, as perhaps there are some subcases we can address (or might be addressed > > by using DTrace with the Analyzer) without having to deal with (a-d). > > The very basic "dtrace -F" output displaying all functions entered/exited under > some one specified function - nothing complex, just "where did my code go?"What code are we talking about here? Are you developing a third-party piece of in-kernel software? For our part, we have very, very little inlining in the Solaris kernel: we either use the preprocessor to define a macro, or have a full-blown function. In our experience, inlining is rarely worth it in C for anything but the smallest functions because it bloats your I-cache footprint. And for those teensy functions where inlining is a win, it usually makes more sense to use the preprocessor... - Bryan -------------------------------------------------------------------------- Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc
Bryan Cantrill said> What code are we talking about here? Are you > developing a third-party > piece of in-kernel software?I''ll reply in a little more detail privately, as it''s proprietary info, but it''s existing kernel code from an ISV - and the inlines are already there. - Jeremy This message posted from opensolaris.org
Jeremy,> Bryan Cantrill said > > What code are we talking about here? Are you > > developing a third-party > > piece of in-kernel software? > > I''ll reply in a little more detail privately, as it''s proprietary info, but > it''s existing kernel code from an ISV - and the inlines are already there.Fair enough. It will help us to know details, even if those details must remain confidential; we haven''t run into this before, and it would be interesting to know which ISV is doing this and why. Depending on the ISV, we might be able to get them to not lean so hard on inlining if only for debuggability''s sake. Of course, if the ISV is NVIDIA or its obfuscating ilk, all bets are off. ;) - Bryan -------------------------------------------------------------------------- Bryan Cantrill, Solaris Kernel Development. http://blogs.sun.com/bmc