dtrace discuss - Sep 2008 - dtrace missing ''unlinkat''? showing process stack?

everyone,

Just out of curiosity, I did a 

dtrace -n ''syscall:::entry { @num[execname, probefunc] = count();
}''

and looked at the entries produced by ''rm''.

I see everything that rm did, *except* the unlinkat - which is unfortunate
because I want to trace which processes have deleted which files.

So - does dtrace contain unlinkat as a probe for a system call? 

Also - how could you integrate the output of something like ptree into the above
call for dtrace, so instead of showing just the pid for the process doing the
system call, you could show the process stack (and if necessary the arguments to
each level in the stack)?

Thanks -

Paul
--
This message posted from opensolaris.org

On Mon, Sep 29, 2008 at 05:26:43PM -0700, Paul Macknee
wrote:
> Just out of curiosity, I did a 
> 
> dtrace -n ''syscall:::entry { @num[execname, probefunc] = count();
}''
> 
> and looked at the entries produced by ''rm''.
> 
> I see everything that rm did, *except* the unlinkat - which is unfortunate
> because I want to trace which processes have deleted which files.
> 
> So - does dtrace contain unlinkat as a probe for a system call? 
Unfortunately, and for anacronistic reasons, the *at system calls were all
implemented as subcodes of the fsat system call:

http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/syscall/fsat.c

Accordingly, you can use the syscall::fsat:entry probe with a predicate on
arg0 to sniff out the calls to unlinkat(2).
> Also - how could you integrate the output of something like ptree into the
> above call for dtrace, so instead of showing just the pid for the process
> doing the system call, you could show the process stack (and if necessary
the
> arguments to each level in the stack)?
You can''t do it from probe context, but you can use the system() action
to
run arbitrary commands. Just remember that those commands run some time after
the probe has fired so you may want to stop() the process:

syscall::fsat:entry
/arg0 == 5/
{
	stop();
	system("ptree %d", pid);
	system("prun %d", pid);
}

Adam

-- 
Adam Leventhal, Fishworks                     http://blogs.sun.com/ahl

ok, 

Thanks for the info - I guess this sort of is asking for it, but exactly how did
you know that 5 was the right number to grep for, aside from knowing the kernel?

Also, isn''t stop, being a destructive actions, not allowed without
dtrace_kernel privilege? Shouldn''t I - as a user, be able to add
destructive actions that modify my own processes, and nobody else''s?

To the first point, IMO, there should be an ''intelligent''
copyinstr - one that is able to either dereference if needed, and one that
converts common numbers like the above into human-readable actions. I''d
love to be able to say:

dtrace -n ''syscall:::entry { @[ execname, pid, probefunc,
copyinstr(arg0, arg1, arg2) ] = count(); }''

and not have to worry whether a given function had 3 arguments or 2 or 1 -- that
it would do the right thing and basically give me human readable output for all
system calls on the system without need for lots of programming. (the other
place this would be helpful for is door calls. I''m still not sure how
to trace past these.)

As for the second point, IMO there has got to be a level between dtrace_user and
dtrace_kernel, perhaps 2 - one where you can effect (ie: have destructive
actions) on your own processes, and one where you can effect and/or trace
processes in your own groups - often, the functions I need to trace past are
ones done by, say a web server, and I don''t want to need sudo or root
privilege to do this. Just MO.


Thanks again,

Paul

(PS: can you write user-based dtrace functions? that would be a quick fix for
the intelligent copyinstr feature.. and would come in helpful in a myriad of
ways..)
--
This message posted from opensolaris.org

Hey Paul,
> Thanks for the info - I guess this sort of is asking for it, but  
> exactly how did you know that 5 was the right number to grep for,  
> aside from knowing the kernel?
Take a look a that link in my last message -- it''s the source code for
the
fsat system call. You''ll see that code 5 is unlinkat(2). As I said  
before,
this state really isn''t ideal; it would be great if we had something a
bit
better for this type of system call (getcontext(2) and setcontext(2) are
other examples of system calls subcodes that share a single system  
call).
> Also, isn''t stop, being a destructive actions, not allowed without
> dtrace_kernel privilege? Shouldn''t I - as a user, be able to add  
> destructive actions that modify my own processes, and nobody
else''s?
Take a look at the chapter on DTrace privileges:

     http://wikis.sun.com/display/DTrace/Security

Note that stop() can be applied to processes to which you have  
sufficient
privileges to modify.
> To the first point, IMO, there should be an ''intelligent''
copyinstr
> - one that is able to either dereference if needed, and one that  
> converts common numbers like the above into human-readable actions.  
> I''d love to be able to say:
>
> dtrace -n ''syscall:::entry { @[ execname, pid, probefunc,  
> copyinstr(arg0, arg1, arg2) ] = count(); }''
>
> and not have to worry whether a given function had 3 arguments or 2  
> or 1 -- that it would do the right thing and basically give me human  
> readable output for all system calls on the system without need for  
> lots of programming. (the other place this would be helpful for is  
> door calls. I''m still not sure how to trace past these.)
Let me see if I understand your request: basically, you want some way to
say "print out all the arguments to this probe formatted in a way that
makes some sense" -- is that right? It''s a reasonable request, but
I
don''t
think we''ll be implementing it any time soon. Perhaps this would be an
opportunity for a third party to build some layered software to  
implement
your suggestion. That is, one could write a program that consumed CTF  
data
to generate a D script that did what you described.
> As for the second point, IMO there has got to be a level between  
> dtrace_user and dtrace_kernel, perhaps 2 - one where you can effect  
> (ie: have destructive actions) on your own processes, and one where  
> you can effect and/or trace processes in your own groups - often,  
> the functions I need to trace past are ones done by, say a web  
> server, and I don''t want to need sudo or root privilege to do
this.
> Just MO.
Take a look at the docs and let me know if you''re still unhappy with
the
privileges we''ve provided. I think you''ll find that the
situation is
pretty
much as you desire.
> (PS: can you write user-based dtrace functions? that would be a  
> quick fix for the intelligent copyinstr feature.. and would come in  
> helpful in a myriad of ways..)
Nope. There''s nothing like that.

Adam

--
Adam Leventhal, Fishworks                        http://blogs.sun.com/ahl

> Take a look at the chapter on DTrace privileges:
> http://wikis.sun.com/display/DTrace/Security
> Note that stop() can be applied to processes to which you have sufficient
> privileges to modify.
Did this change in recent versions of dtrace? I''m not seeing this work:

    syscall::fsat:entry /arg0 == 5/ 
    {
        stop();
        printf("%d %s %d %s\n", pid, execname, arg0, copyinstr(arg1));
        system("ptree %d", pid);  system("prun  %d", pid);
    }

    dtrace: script ''/tmp/del.d'' matched 1 probe
    dtrace: could not enable tracing: Destructive actions not allowed
> let me see if I understand your request: basically, you want some way to
> say "print out all the arguments to this probe formatted in a way that
> makes some sense" -- is that right? 
Basically, but let me put it a bit differently. I want the ''new
copyinstr'' to be able to use introspection on the probefunc to
determine how to treat the given arguments and to transform them into a format
that becomes human readable (if possible).

I cannot stress how much easier that this would make dtrace to use. Every single
major scripting language out there has a ''dumper'' function
that can intelligently parse data structures (even gdb has the ability); dtrace
should be no exception. For system calls, If the arg array had the probefunc
attached to it as a property, then you could pass it to such a function and have
a lookup table on how to treat the arguments.

A pointer to the arg array would help too (does it exist?) so you could pass the
entire array to this function; the ability to stringify data structures behind
args would help as well.
> Perhaps this would be an opportunity for a third party to build some
layered
> software to implement your suggestion. That is, one could write a program 
> that consumed CTF data to generate a D script that did what you described.
Great, I''m a third party.. ;) From my point of view, I think that
generating it from headers might be easier, especially if I could program it in
a dtrace function of my own design, and just look at the headers i''m
most interested in.

Come to think of it - exactly how could you scale up such a generated program
that doesn''t use user function calls? Suppose I wanted to use the
intelligent copystr in my own program in a myriad of places - are you really
suggesting that having an interpreter which takes user-level functions and
expands them in my source code is the best way to do it?

I think you guys should just bite the bullet and make user-level functions.. I
don''t think you''d even need to make kludgy syntax for it -
just inherit from python or perl:

user::intelligent_copystr
{
    printf ("%a", @args);
}

Thanks much,

Paul
--
This message posted from opensolaris.org

Paul Macknee wrote:
>> Take a look at the chapter on DTrace privileges:
> 
>> http://wikis.sun.com/display/DTrace/Security
> 
>> Note that stop() can be applied to processes to which you have
sufficient
>> privileges to modify.
> 
> Did this change in recent versions of dtrace? I''m not seeing this
work:
> 
>     syscall::fsat:entry /arg0 == 5/ 
>     {
>         stop();
>         printf("%d %s %d %s\n", pid, execname, arg0,
copyinstr(arg1));
>         system("ptree %d", pid);  system("prun  %d",
pid);
>     }
> 
>     dtrace: script ''/tmp/del.d'' matched 1 probe
>     dtrace: could not enable tracing: Destructive actions not allowed
the error message is telling you exactly what''s wrong - you need to
enable
destructive operations. This is a DTrace issue, not one of permission.
see the -w option in the man page.

Michael
-- 
Michael Schuster        http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''

On Sep 30, 2008, at 8:03 PM, Paul Macknee wrote:
>> Perhaps this would be an opportunity for a third party to build  
>> some layered
>> software to implement your suggestion. That is, one could write a  
>> program
>> that consumed CTF data to generate a D script that did what you  
>> described.
>
> Great, I''m a third party.. ;) From my point of view, I think that
> generating it from headers might be easier, especially if I could  
> program it in a dtrace function of my own design, and just look at  
> the headers i''m most interested in.
Why would you want to do this as a "dtrace function"? This seems like
a
preprocessing step that you''d want to do statically rather than at  
runtime.

If DTrace were to support functionality of a "dumper" as you suggest,
it
would do so by statically examining the enabled probes and emitting the
appropriate tracing commands.
> Come to think of it - exactly how could you scale up such a  
> generated program that doesn''t use user function calls? Suppose I
> wanted to use the intelligent copystr in my own program in a myriad  
> of places - are you really suggesting that having an interpreter  
> which takes user-level functions and expands them in my source code  
> is the best way to do it?
Potentially, but it depends on what you want.
> I think you guys should just bite the bullet and make user-level  
> functions..
That''s a pretty complicated suggestion, and one that''s
unlikely to be
possible in the way that you''re imagining.

There''s an existing RFE for an equivalent to mdb''s ::print
functionality
that would pretty-print a datum according to its time. This would  
perhaps
be a step towards what you''re suggesting.

Adam

--
Adam Leventhal, Fishworks                        http://blogs.sun.com/ahl

On Tue, Sep 30, 2008 at 08:46:23PM -0700, Adam Leventhal
wrote:
> There''s an existing RFE for an equivalent to mdb''s
::print functionality
> that would pretty-print a datum according to its time. This would  
> perhaps
> be a step towards what you''re suggesting.
I could see something like:

pid$1:libsomelibrary:some_function:entry
{
	self->some_item =
copyin_expr("arg[0]->field1->field2->field3");
	...
}

and the D *compiler* looking up the CTF for this and figuring out what
sequence of copyins to translace that into, including, possibly,
defining a useful struct type for self->some_item.

The bitness of the target would have to be known at compile-time, else
the compiler would have to generate DOF code for both, 32- and 64-bit
targets and branch at run-time according to which the running target is.

In fact, this wouldn''t be very hard to do as a script layered atop
DTrace.  One could start with John Levon''s wonderful ctftoh program and
go from there.

Oh, BTW, it''d be nice if we built executables with CTF too...

(ctftoh was useful to me recently in manually writing some tricky
copyins with pointer dereferences.)

Nico
--

> Why would you want to do this as a "dtrace function"? This seems
like a
> preprocessing step that you''d want to do statically rather than at
> runtime.
Suppose I want to make a simple function - a ''truss lite'' as
it were - which does an aggregation of all system calls and their arguments, in
human readable form. With a dumper-like function - and an expression which
expands all arguments, you could say:

syscall:::entry
{
     @[ execname, probefunc, stringify(probefunc, @arg) ] = count();
}

In stringify - if I could write it - I could keep in one place all of the
functionality for parsing, ignoring, or passing through the arguments that come
into the syscall.

I could then reuse that functionality in other places - say in an printf, which
I could use in any probe, or in other scripts.

The alternative I see, is to need to know the usage about each syscall (all
hundreds of them) and write hundreds of separate probes that know how the
arguments are to be parsed.  As it is, if I even try to do intelligent
preprocessing:

syscall:::entry
{
    @[ execname, probefunc, copyinstr(arg0)] = count();
}

this complains loudly (as it should) about dereferencing errors.

I''m open to suggestions on how to write something like the above using
the current  dtrace - so - how would you do it using the current dtrace, using
your ''preprocessing'' method, and assuming that I want to reuse
the functionality, potentially, in other scripts?

Thanks,

Paul
--
This message posted from opensolaris.org

On Wed, Oct 01, 2008 at 12:16:07AM -0500, Nicolas Williams
wrote:
> pid$1:libsomelibrary:some_function:entry
> {
> 	self->some_item =
copyin_expr("arg[0]->field1->field2->field3");
> 	...
> }
> 
> and the D *compiler* looking up the CTF for this and figuring out what
> sequence of copyins to translace that into, including, possibly,
> defining a useful struct type for self->some_item.
> 
> The bitness of the target would have to be known at compile-time, else
> the compiler would have to generate DOF code for both, 32- and 64-bit
> targets and branch at run-time according to which the running target is.
We don''t even need the copyin_expr() part -- there''s no reason
we couldn''t
"know" that copyins were implicitly required.

Adam

-- 
Adam Leventhal, Fishworks                     http://blogs.sun.com/ahl

On Wed, Oct 01, 2008 at 12:02:04PM -0700, Paul Macknee
wrote:
> The alternative I see, is to need to know the usage about each syscall (all
> hundreds of them) and write hundreds of separate probes that know how the
> arguments are to be parsed.  As it is, if I even try to do intelligent
> preprocessing:
That would already be necessary: there''s no type information for system
calls
and no typed arguments. For a DTrace-based truss you''d need to
replicate the
gigantic tables in the truss source code.

Adam


-- 
Adam Leventhal, Fishworks                     http://blogs.sun.com/ahl

On Wed, Oct 01, 2008 at 12:59:15PM -0700, Adam Leventhal
wrote:
> On Wed, Oct 01, 2008 at 12:16:07AM -0500, Nicolas Williams wrote:
> > pid$1:libsomelibrary:some_function:entry
> > {
> > 	self->some_item =
copyin_expr("arg[0]->field1->field2->field3");
> > 	...
> > }
> > 
> > and the D *compiler* looking up the CTF for this and figuring out what
> > sequence of copyins to translace that into, including, possibly,
> > defining a useful struct type for self->some_item.
> > 
> > The bitness of the target would have to be known at compile-time, else
> > the compiler would have to generate DOF code for both, 32- and 64-bit
> > targets and branch at run-time according to which the running target
is.
> 
> We don''t even need the copyin_expr() part -- there''s no
reason we couldn''t
> "know" that copyins were implicitly required.
Oh, I know, I didn''t mean that it''d have to look like a
function.

Native compiler support might look completely different, better
integrated, something like

pid$1:libsomelibrary:some_function:entry
{
    self->some_item = arg[0]->field1->field2->field3;
    trace(self->some_item.field4);
    ...
}

If I were building a pre-processor to generate D code to replace such an
expression I''d probably just add a copyin directives section between a
probe''s predicate and actions.  That would make it easy to extract the
copyin expressions without having to parse the actions.

On Wed, Oct 01, 2008 at 01:00:52PM -0700, Adam Leventhal
wrote:
> On Wed, Oct 01, 2008 at 12:02:04PM -0700, Paul Macknee wrote:
> > The alternative I see, is to need to know the usage about each syscall
(all
> > hundreds of them) and write hundreds of separate probes that know how
the
> > arguments are to be parsed.  As it is, if I even try to do intelligent
> > preprocessing:
> 
> That would already be necessary: there''s no type information for
system calls
> and no typed arguments. For a DTrace-based truss you''d need to
replicate the
> gigantic tables in the truss source code.
Usually there''s a kernel function that gets all the copied-in arguments
of a syscall (e.g., copen(), in the case of open(), open64(),
openat()...), so judicious use of the FBT provider can get you what you
want.  But that''s not stable.  It''d be nice if for every
syscall there
was a stable function that got the copied-in args.

Nico
--