dtrace discuss - Aug 2009 - Problem with missing custom dtrace probes [71425968]

Hi all,

I have a customer who has a strange problem with building DTrace probes
into their own code.  They describe it as:
> We are seeing issues with custom dtrace probes we have added to our
> application code, where some of the probe points we have defined are
> not present; see below for details.  Can you please look into this
> problem and suggest a next step (or let us know if there are any
> limitations in this area that we should be aware of). 
>  
>   Background   
>  
> Our application code includes many (around 90 or so) libraries which
> are dynamically loaded from the main executable with a dlopen() call
> at start of day.
>  
> We have defined a dtrace provider which defines several custom dtrace
> probes we use in our application code.  The .d file for this provider
> and its probes is included in the makefiles for each of our
> application''s libraries.  Each library typically includes multiple
> instances of the dtrace probe (using the DTRACE_PROBE<n> macros).
>  
>   Issue
>  
> We are seeing that only some of the probe points for these custom
> probes are accessible to dtrace scripts.  Listing the probes using
> "dtrace -l -P <provider name> <pid>" only shows all
the probe types,
> but only a subset of the probe points that we''d expect to see.
>  
>       * There is no clear pattern as to which libraries do have the
>         probe points which we''d expect and which do not. 
>       * We build different configurations of our application code in
>         which a different subset of our libraries are loaded.
>         Comparing the probe points from different configurations shows
>         that we''re not always missing the same probe points - i.e.
>         configurations A and B both include library x, but x has the
>         correct probe points in configuration A but not in
>         configuration B.  As both configurations use the same library,
>         this suggests there isn''t anything wrong with particular
>         libraries, but rather that we are hitting some sort of limit
>         at runtime. 
I asked them for a small test case, and they can''t rip something out,
only provide a section of their code, which they''d prefer to keep that
under NDA, so I can''t post it to this alias.

Can someone help us figure this out?

Ta,
Mark

Without knowing more about the code, it''s difficult to be sure, but  
this is an issue that we experienced in the MySQL code and which we  
had to work hard to try and eliminate.

We now jump through a few hoops to achieve what we want across a  
variety of libraries and possible build types.

Feel free to contact me at mc.brown at sun.com and I will be in touch  
when I return properly next week with more detail, but I suspect this  
is fixable.

MC

On 4 Aug 2009, at 16:45, Mark R. Bowyer wrote:
> Hi all,
>
> I have a customer who has a strange problem with building DTrace  
> probes
> into their own code.  They describe it as:
>
>> We are seeing issues with custom dtrace probes we have added to our
>> application code, where some of the probe points we have defined are
>> not present; see below for details.  Can you please look into this
>> problem and suggest a next step (or let us know if there are any
>> limitations in this area that we should be aware of).
>>
>> Background
>>
>> Our application code includes many (around 90 or so) libraries which
>> are dynamically loaded from the main executable with a dlopen() call
>> at start of day.
>>
>> We have defined a dtrace provider which defines several custom dtrace
>> probes we use in our application code.  The .d file for this provider
>> and its probes is included in the makefiles for each of our
>> application''s libraries.  Each library typically includes
multiple
>> instances of the dtrace probe (using the DTRACE_PROBE<n> macros).
>>
>> Issue
>>
>> We are seeing that only some of the probe points for these custom
>> probes are accessible to dtrace scripts.  Listing the probes using
>> "dtrace -l -P <provider name> <pid>" only shows
all the probe types,
>> but only a subset of the probe points that we''d expect to see.
>>
>> * There is no clear pattern as to which libraries do have the
>> probe points which we''d expect and which do not.
>> * We build different configurations of our application code in
>> which a different subset of our libraries are loaded.
>> Comparing the probe points from different configurations shows
>> that we''re not always missing the same probe points - i.e.
>> configurations A and B both include library x, but x has the
>> correct probe points in configuration A but not in
>> configuration B.  As both configurations use the same library,
>> this suggests there isn''t anything wrong with particular
>> libraries, but rather that we are hitting some sort of limit
>> at runtime.
>
> I asked them for a small test case, and they can''t rip something
out,
> only provide a section of their code, which they''d prefer to keep
that
> under NDA, so I can''t post it to this alias.
>
> Can someone help us figure this out?
>
> Ta,
> Mark
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>