dtrace discuss - Nov 2007 - Can you trace a library across all pids?

Hi,

I think I know the answer to this, which I think is no, but want to make 
sure.  Is there any way to trace any invocation of a library (and 
library routine) by any pid?  I want to record all invocations of a call 
to a certain library call, no matter what pid invokes it ultimately. 

Please tell me there is some trickery that will allow me to do this.

pete

On Thu, Nov 01, 2007 at 01:58:29PM -0500, Peter Shoults
wrote:
> I think I know the answer to this, which I think is no, but want to make 
> sure.  Is there any way to trace any invocation of a library (and 
> library routine) by any pid?  I want to record all invocations of a call 
> to a certain library call, no matter what pid invokes it ultimately. 
>
> Please tell me there is some trickery that will allow me to do this.
You can''t do that today with DTrace. Of course, it''s only
software -- we
could implement something like that if the use case was compelling. Can
you explain what you''re doing?

Adam

-- 
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl

Adam Leventhal wrote:
> On Thu, Nov 01, 2007 at 01:58:29PM -0500, Peter Shoults wrote:
>   
>> I think I know the answer to this, which I think is no, but want to
make
>> sure.  Is there any way to trace any invocation of a library (and 
>> library routine) by any pid?  I want to record all invocations of a
call
>> to a certain library call, no matter what pid invokes it ultimately. 
>>
>> Please tell me there is some trickery that will allow me to do this.
>>     
>
> You can''t do that today with DTrace. Of course, it''s only
software -- we
> could implement something like that if the use case was compelling. Can
> you explain what you''re doing?
>
> Adam
>
>   
Adam,

Before I replied, I wanted to make sure I could discuss this on the 
opensolaris aliases.  The code I am working with is Kerberos code that 
we have license to work with and is in the public domain, so I am 
cleared to go.

The overall problem:

Data corruption of an internal kerberos database was experienced, and 
analysis of existing data did not yield any conclusive root cause to the 
problem. 

Next step action plan:

We wanted to develop a dtrace script that would capture the raw data 
that would be used to modify the internal database just before it was 
actually used to update the database.   I have written this script, and 
it does dump out the record.  However, it turns out that the database 
can be modified by three different methods, and each of these can be run 
many times under different pid numbers of course.  The three methods are 
kadmin, kadmin.local and kpasswd.

So, while my dtrace script did dump out exactly what we wanted, there 
was no way to get it to track and dump out records for multiple 
different pids that could be started up at any time.

Hope that helps provide a good indication of what we were trying to 
accomplish.

Peter

Hey Peter,
> Before I replied, I wanted to make sure I could discuss this on the 
> opensolaris aliases.  The code I am working with is Kerberos code that 
> we have license to work with and is in the public domain, so I am 
> cleared to go.
> 
> The overall problem:
> 
> Data corruption of an internal kerberos database was experienced, and 
> analysis of existing data did not yield any conclusive root cause to the 
> problem. 
> 
> Next step action plan:
> 
> We wanted to develop a dtrace script that would capture the raw data 
> that would be used to modify the internal database just before it was 
> actually used to update the database.   I have written this script, and 
> it does dump out the record.  However, it turns out that the database 
> can be modified by three different methods, and each of these can be run 
> many times under different pid numbers of course.  The three methods are 
> kadmin, kadmin.local and kpasswd.
Thanks for the use case!  One thing to investigate:  add an SDT probe in
the common library where the database is modified.  Especially with
is-enabled probes, you can construct this such that it provides rich
arguments (e.g., the nature of the query or transaction) -- and then the
ability to dynamically instrument all processes that make the call just
falls out.  

And to be honest, it''s such a nasty problem to allow instrumentation of
arbitrary functions across arbitrarily many (and arbitrarily dynamic)
processes that we''re unlikely to solve it (at least in the near future)
--
especially with SDT probes providing such a reasonable solution...

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Sun Microsystems FishWorks.       http://blogs.sun.com/bmc

On Wed, Nov 21, 2007 at 09:58:49AM -0800, Bryan Cantrill
wrote:
> > We wanted to develop a dtrace script that would capture the raw data 
> > that would be used to modify the internal database just before it was 
> > actually used to update the database.   I have written this script,
and
> > it does dump out the record.  However, it turns out that the database 
> > can be modified by three different methods, and each of these can be
run
> > many times under different pid numbers of course.  The three methods
are
> > kadmin, kadmin.local and kpasswd.
> 
> Thanks for the use case!  One thing to investigate:  add an SDT probe in
> the common library where the database is modified.  Especially with
> is-enabled probes, you can construct this such that it provides rich
> arguments (e.g., the nature of the query or transaction) -- and then the
> ability to dynamically instrument all processes that make the call just
> falls out.  
> 
> And to be honest, it''s such a nasty problem to allow
instrumentation of
> arbitrary functions across arbitrarily many (and arbitrarily dynamic)
> processes that we''re unlikely to solve it (at least in the near
future) --
> especially with SDT probes providing such a reasonable solution...
I''ve wanted that myself, but it does seem like a difficult problem!

One way to deal with this particular use case might be to use the
syscall provider to catch opens for write of that DB, stop caller and
start a new instance of dtrace (via system()) that traces the stopped
process and pruns it.  I''ve done this sort of thing before and it works
OK.  It slows down the DB open operation a lot (since system() involves
fork/execing), and it requires destructive actions, but it''s livable.

Nico
--

> We wanted to develop a dtrace script that would capture the raw data 
> that would be used to modify the internal database just before it was 
> actually used to update the database.   I have written this script, and 
> it does dump out the record.  However, it turns out that the database 
> can be modified by three different methods, and each of these can be run 
> many times under different pid numbers of course.  The three methods are 
> kadmin, kadmin.local and kpasswd.
> 
> So, while my dtrace script did dump out exactly what we wanted, there 
> was no way to get it to track and dump out records for multiple 
> different pids that could be started up at any time.
One common technique is to write a D script which watches for new processes
to be spawned, and then invokes another D script on that new process. In
other words, something like this:

proc:::start
/execname == "bash"/
{
	stop();
	system("dtrace -s foo.d -p %d", pid);
}

foo.d:
BEGIN
{
	system("prun %d", $target);
}

...

Might that work for this problem?

Adam

-- 
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl