dtrace discuss - Apr 2006 - DTrace as a security tool / http://systrace.org

I''d like to see if we can use DTrace to as the kernel implementation of
the BSD systrace security policy system (http://www.systrace.org).  I 
don''t really want to port systrace to Solaris because I think with 
DTrace we already have all the necessary in kernel hooks to do this.

With systrace you express things like: "httpd can bind to port 80 but 
not any other port, it can open for writting files in /var/apache/log 
but no where else".

It seems that a lot of what systrace needs to implement the policy is 
already available with the fbt and syscall providers since it largely
just "interposes" on system calls.

I think the key thing here is we would need for DTrace not just observe 
but to be able to change program flow, by that I mean if the policy says 
that a given system call isn''t allowed we need to return in error 
without ever running the code of the system call.  As I understand it we 
can''t do that in DTrace today - or am I missing something.

While part of me things that systrace requires far too much 
implementation detail to be coded into the "policies" I also think it
is
a useful tool.

Opinions ?

-- 
Darren J Moffat

I''m assuming you want something very fine-grained,
and that the capabilities Solaris gained from Trusted
Solaris aren''t where you want to go...

  If so,  suspect you''d want to literally interpose on 
the system calls on Solaris.  It''s quite easy to
build library interposers,  I have a shell script
that generates skeletons like the following
(somewhat flattened by the forum):

#include <dlfcn.h>      /* For dlopen(). */
#include <link.h>
#include <assert.h>     /* For assert() macro. */
/*
 * pthread_mutex_lock -- intercept pthread_mutex_lock
 */
int pthread_mutex_lock(pthread_mutex_t *mutex){
        static void *actual_pthread_mutex_lock = NULL;

        if (actual_pthread_mutex_lock == NULL) {
                actual_pthread_mutex_lock = dlsym(RTLD_NEXT,
"pthread_mutex_lock");
                assert(actual_pthread_mutex_lock != NULL);
        }
        return ((int (*)(pthread_mutex_t *))actual_pthread_mutex_lock)(mutex);
}

--dave
 
 
This message posted from opensolaris.org

For some context yes I do want very fine-grained control and I''m more
that familiar with what Trusted Solaris does since I''ve been a user of
it for about 11 years and do code and design reviews for the team!

Doing this in a library isn''t the answer because it is too easy to
bypass library level interposing from inside the application code and/or just by
changing the environment.  I really needs to be done in the kernel so that it
applies everywhere and isn''t possible to by pass it.

The systrace.org stuff I pointed to is a kernel level policy enforcement system
with a userland config and "feedback" mechanism.  Thats basically what
I''m suggesting.

I''m trying to find out from DTrace experts if they thing it is possible
to use the DTrace infrastructure to do the kernel parts of what systrace does. 
This might mean just using the kernel parts of DTrace and having another
libdtrace consumer in userland that parses the systrace policy language and
builds the compiled D stuff for pushing to the kernel to enforce the policy.
 
 
This message posted from opensolaris.org

G''Day Darren,

On Fri, 7 Apr 2006, Darren J. Moffat wrote:
[...]
> The systrace.org stuff I pointed to is a kernel level policy
> enforcement system with a userland config and "feedback"
mechanism.
> Thats basically what I''m suggesting.
Yes, DTrace has the userland config and feedback mechanisms.
> I''m trying to find out from DTrace experts if they thing it is
possible
> to use the DTrace infrastructure to do the kernel parts of what systrace
> does.  This might mean just using the kernel parts of DTrace and having
> another libdtrace consumer in userland that parses the systrace policy
> language and builds the compiled D stuff for pushing to the kernel to
> enforce the policy.
The D-Team can answer this better than me; but here are some thoughts:

This isn''t something DTrace appears to be designed for. That''s
not to say
DTrace can''t do it, it may just need nudging and some careful thought.

Having some efficient (destructive) functions to block syscalls would be
(I would guess) not a huge addition to the DTrace kernel framework.

A greater problem may the idea that DTrace can be run as a critical
daemon to enforce security; DTrace is designed with saftey-valves - for
example, it will abort if it is consuming too much CPU. This is great for
an analysis tool, this sucks for a security framework. Each saftey valve
is a convienent target to crackers. They would need to be turned off
(which I believe can be done with some mdb -kw jiggery-pokery - but
pretend I didn''t say that), and so we''d need to think long and
hard about
the implications of disabling those saftey-valves.

A more positive thought - if this were to be done, it wouldn''t be
inconceivable that a new libdtrace based daemon could be written that
would enforce systrace like policies.

cheers,

Brendan

Brendan Gregg wrote:
> This isn''t something DTrace appears to be designed for.
That''s not to say
> DTrace can''t do it, it may just need nudging and some careful
thought.
That was my thought too.
> Having some efficient (destructive) functions to block syscalls would be
> (I would guess) not a huge addition to the DTrace kernel framework.
> A more positive thought - if this were to be done, it wouldn''t be
> inconceivable that a new libdtrace based daemon could be written that
> would enforce systrace like policies.
Yep I think you''ve understood what I''m after, using what
DTrace has
provided so far and building on it in userland and in kernel to make a 
systrace implementation out of it.

The "easy" part is probably the userland parsing of the systrace.org 
style policy files and translating them into DIF.  I think the hard part 
is the extension on the kernel side so that it is always present and 
completely synchronous.  This might not be the best way to implement 
what I''m trying to do, it might be better to make policy.c pluggable
and
add more secpolicy_*() calls all over the place but it just seems that 
systrace is so close in some areas to even the simple fbt provider.

Thanks Brendan for the input.

-- 
Darren J Moffat

On Fri, Apr 07, 2006 at 11:25:18PM +1000, Brendan Gregg
wrote:
> G''Day Darren,
> 
> On Fri, 7 Apr 2006, Darren J. Moffat wrote:
> [...]
> > The systrace.org stuff I pointed to is a kernel level policy
> > enforcement system with a userland config and "feedback"
mechanism.
> > Thats basically what I''m suggesting.
> 
> Yes, DTrace has the userland config and feedback mechanisms.
> 
> > I''m trying to find out from DTrace experts if they thing it
is possible
> > to use the DTrace infrastructure to do the kernel parts of what
systrace
> > does.  This might mean just using the kernel parts of DTrace and
having
> > another libdtrace consumer in userland that parses the systrace policy
> > language and builds the compiled D stuff for pushing to the kernel to
> > enforce the policy.
> 
> The D-Team can answer this better than me; but here are some thoughts:
> 
> This isn''t something DTrace appears to be designed for.
That''s not to say
> DTrace can''t do it, it may just need nudging and some careful
thought.
Brendan''s right, and I would actually go a step further:  not only does
this problem lie outside of the design center for DTrace, it actually
runs contrary to it.  DTrace is designed to preserve the integrity of the 
system over its own operation.  One manifestation of this is the deadman
mechanism that Brendan mentioned; another is DTrace''s decision to drop
records if any sort of fault or error is encountered.  This is absolutely
relevant for what you''d like to do, because you want to (for example)
conditionally fail system calls based on the contents pointed to by
arguments.  In DTrace, failure to examine those contents due to a page fault
will cause the clause to abort -- but if the page is from a valid mapping
(and it just needs to be faulted in), this is absolutely _not_ the behavior
you want.  Moreover, the amount of infrastructure that you''re
leveraging
from DTrace is pretty tiny -- the system call interposition mechanism which
you seek to reuse took just a couple of hours to implement.  I think what
you need is another system call interposer -- which means that you''ll
need
to architect and implement a general system call interposition mechanism,
for which the syscall provider will ultimately become a client.  This would
probably have value outside of just your work:  there are some ISVs today
that develop similar security/auditing products that currently
directly butcher the system call table (and thereby violate the integrity
of the system in ways they don''t understand); it would be at least a
step
in the right direction for these ISVs to have a stable way of interposing
on the system call table...

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

Thanks Bryan for the informative response, I was starting to suspect 
that was how it would be seen.

Yes a general purpose syscall interposer is certainly needed, I got an 
email from a colleague in Spain about an ISV wanting to port a security 
product over from Linux to OpenSolaris and one of the many things it 
does is syscall butchery.

Thanks again, I''ll take this back to the security world for now and
I''ll
be back to DTrace if/when we get a generic syscall interposer that the 
DTrace provider can use.

Darren

On Wed, Apr 05, 2006 at 02:12:25PM +0100, Darren J Moffat
wrote:
> Opinions ?
Add a few additional actions and it could be done, but I worry about
robustness, since DTrace can drop events under pressure...

On Wed, Apr 05, 2006 at 11:36:30AM -0700, David S. Collier-Brown
wrote:
> I''m assuming you want something very fine-grained,
> and that the capabilities Solaris gained from Trusted
> Solaris aren''t where you want to go...
You must be referring to Solaris privileges (which are rather different
from Linux or Trusted Solaris 8 capabilities).  See privileges(5).

And, yes, that''s what I think Darren means (we''ve talked about
this
before :)
>   If so,  suspect you''d want to literally interpose on 
> the system calls on Solaris.  It''s quite easy to
> build library interposers,  I have a shell script
> that generates skeletons like the following
> (somewhat flattened by the forum):
I think Darren wants something that doesn''t require interposition.

Nico
--

On Fri, Apr 07, 2006 at 11:51:17AM -0500, Nicolas Williams
wrote:
> On Wed, Apr 05, 2006 at 02:12:25PM +0100, Darren J Moffat wrote:
> > Opinions ?
> 
> Add a few additional actions and it could be done, but I worry about
> robustness, since DTrace can drop events under pressure...
I should have read the whole thread first...

>Brendan''s right, and I would actually go a step further:  not only
does
>this problem lie outside of the design center for DTrace, it actually
>runs contrary to it.  DTrace is designed to preserve the integrity of the 
>system over its own operation.  One manifestation of this is the deadman
>mechanism that Brendan mentioned; another is DTrace''s decision to
drop
>records if any sort of fault or error is encountered.  This is absolutely
>relevant for what you''d like to do, because you want to (for
example)
>conditionally fail system calls based on the contents pointed to by
>arguments.  In DTrace, failure to examine those contents due to a page fault
>will cause the clause to abort -- but if the page is from a valid mapping
>(and it just needs to be faulted in), this is absolutely _not_ the behavior
>you want.  Moreover, the amount of infrastructure that you''re
leveraging
>from DTrace is pretty tiny -- the system call interposition mechanism which
>you seek to reuse took just a couple of hours to implement.  I think what
>you need is another system call interposer -- which means that
you''ll need
>to architect and implement a general system call interposition mechanism,
>for which the syscall provider will ultimately become a client.  This would
>probably have value outside of just your work:  there are some ISVs today
>that develop similar security/auditing products that currently
>directly butcher the system call table (and thereby violate the integrity
>of the system in ways they don''t understand); it would be at least
a step
>in the right direction for these ISVs to have a stable way of interposing
>on the system call table...

I''m actually trying to design a mechanism which intervenes in the
centralized Solaris policy.

The idea is that in general, privileges are too coarse and that you may
at times want things like "read this file" rather than
"file_dac_read"
or "bind to this port" rather than "net_privaddr".

The natural way to handle this, IMHO, is to propagate such policy
violations to a higher order arbiter which may make a different decision.

So a polciy failure in effect will grow a "defer to higher authority"
rather than "return errno".

Casper

>On Wed, Apr 05, 2006 at 11:36:30AM -0700, David S. Collier-Brown wrote:
>> I''m assuming you want something very fine-grained,
>> and that the capabilities Solaris gained from Trusted
>> Solaris aren''t where you want to go...
>
>You must be referring to Solaris privileges (which are rather different
>from Linux or Trusted Solaris 8 capabilities).  See privileges(5).
They are called privilges in Trusted Solaris also.

It is amusing, or perhaps sad, how "privileges" got to be renamed
"capabilities" in one of the last drafts before the standard died.
And why?  Because "privileges" were felt to have some other
connotations
in the real world.  It was then, of course, real smart to
redefine an *existing* term from the computer industry into meaning
something completely different.

Casper

On Fri, Apr 07, 2006 at 07:09:39PM +0200, Casper.Dik at Sun.COM
wrote:
> I''m actually trying to design a mechanism which intervenes in the
> centralized Solaris policy.
> 
> The idea is that in general, privileges are too coarse and that you may
> at times want things like "read this file" rather than
"file_dac_read"
> or "bind to this port" rather than "net_privaddr".
> 
> The natural way to handle this, IMHO, is to propagate such policy
> violations to a higher order arbiter which may make a different decision.
Which is very much like systrace.

What makes DTrace attractive here is not merely its syscall provider,
but the existing consumer infrastructure and, of course, D.

"Robust" means very different things to DTrace and systrace, thus a
major impedance mismatch.

It might be that we could fix that by treating syscall provider events
more "robustly" in the systrace sense (i.e., don''t drop them)
and by
having user-level DT actions that can cause the thread being traced to
go through all the page faulting needed to get at the syscall arguments,
and so on.  But there may be a lot of resistance to this -- it really is
re-purposing DTrace far afield from its intended purpose, arguably not a
wise thing to do.
> So a polciy failure in effect will grow a "defer to higher
authority"
> rather than "return errno".
Another possibility is that the "higher authority" evaluates the
syscall
before policy checks/access controls are evaluated and elevates the
calling thread''s privileges temporarily.  IIRC this is what systrace
does.

Nico
--

>Which is very much like systrace.
Quite.  But a more centralized implementation.
>What makes DTrace attractive here is not merely its syscall provider,
>but the existing consumer infrastructure and, of course, D.
It is unclear whether the syscall provider is sufficient
or even necessary.
>"Robust" means very different things to DTrace and systrace, thus
a
>major impedance mismatch.
>It might be that we could fix that by treating syscall provider events
>more "robustly" in the systrace sense (i.e., don''t drop
them) and by
>having user-level DT actions that can cause the thread being traced to
>go through all the page faulting needed to get at the syscall arguments,
>and so on.  But there may be a lot of resistance to this -- it really is
>re-purposing DTrace far afield from its intended purpose, arguably not a
>wise thing to do.
Quite; dtrace also depends, perhaps, too much on the implementation
of the kernel, including the syscall provider as the arguments to
system calls are not fixed.

To have a generic filtering interface, you would need to push stability
to a higher level, guarantee no loss *and* we need to be able to
make decisions based at the call site *and* wait until the event
is handled.  I''m not sure "D" would be the most suitable
language
for such policy decisions; specifically since you may want to
make such decisions on the fly, with user interaction.
>> So a polciy failure in effect will grow a "defer to higher
authority"
>> rather than "return errno".
>
>Another possibility is that the "higher authority" evaluates the
syscall
>before policy checks/access controls are evaluated and elevates the
>calling thread''s privileges temporarily.  IIRC this is what
systrace
That''s yucky to the higest degree; the problem is that you need to
predict and perform all you copyin work at the syscall boundary, something
we currently avoid on purpose.

It''s also the expense: I prefer to defer the complete expense of this
exception handling, as this may include an upcall to userland, to the
point that we have decided we really want to fail the call.  Adding
arbitrary expense at the point of failure is not an issue, I think.

Casper

> "Robust" means very different things to DTrace and systrace, thus
a
> major impedance mismatch.
> 
> It might be that we could fix that by treating syscall provider events
> more "robustly" in the systrace sense (i.e., don''t drop
them) and by
> having user-level DT actions that can cause the thread being traced to
> go through all the page faulting needed to get at the syscall arguments,
> and so on.  
What''s a "user-level DTrace action"?  Such a notion is rife
with complexities
that undermine both the design center and architectural ethos of DTrace...
> But there may be a lot of resistance to this -- it really is
> re-purposing DTrace far afield from its intended purpose, arguably not a
> wise thing to do.
I don''t think "resistance" is quite the right word; I think
"refusal" is
more the word you''re looking for.

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

>I don''t think "resistance" is quite the right word; I
think "refusal" is
>more the word you''re looking for.
I feel a OpenSolaris fork is in the air :-)

Casper (ducks)

On Fri, Apr 07, 2006 at 10:42:56AM -0700, Bryan Cantrill
wrote:
> 
> > "Robust" means very different things to DTrace and systrace,
thus a
> > major impedance mismatch.
> > 
> > It might be that we could fix that by treating syscall provider events
> > more "robustly" in the systrace sense (i.e., don''t
drop them) and by
> > having user-level DT actions that can cause the thread being traced to
> > go through all the page faulting needed to get at the syscall
arguments,
> > and so on.  
> 
> What''s a "user-level DTrace action"?  Such a notion is
rife with complexities
> that undermine both the design center and architectural ethos of DTrace...
It''s how system() is implemented, IIRC.
> > But there may be a lot of resistance to this -- it really is
> > re-purposing DTrace far afield from its intended purpose, arguably not
a
> > wise thing to do.
> 
> I don''t think "resistance" is quite the right word; I
think "refusal" is
> more the word you''re looking for.
:)

On Fri, Apr 07, 2006 at 07:57:06PM +0200, Casper.Dik at Sun.COM
wrote:
> 
> >I don''t think "resistance" is quite the right word;
I think "refusal" is
> >more the word you''re looking for.
> 
> I feel a OpenSolaris fork is in the air :-)
Er, not by me...  I want the feature, and though Darren and I have
talked about building it atop DTrace I''m not wedded to that idea, and,
I
think, neither is Darren.  Where''s the fork? :)

Nico
--

On Fri, Apr 07, 2006 at 01:00:14PM -0500, Nicolas Williams
wrote:
> On Fri, Apr 07, 2006 at 10:42:56AM -0700, Bryan Cantrill wrote:
> > 
> > > "Robust" means very different things to DTrace and
systrace, thus a
> > > major impedance mismatch.
> > > 
> > > It might be that we could fix that by treating syscall provider
events
> > > more "robustly" in the systrace sense (i.e.,
don''t drop them) and by
> > > having user-level DT actions that can cause the thread being
traced to
> > > go through all the page faulting needed to get at the syscall
arguments,
> > > and so on.  
> > 
> > What''s a "user-level DTrace action"?  Such a notion
is rife with complexities
> > that undermine both the design center and architectural ethos of
DTrace...
> 
> It''s how system() is implemented, IIRC.
There are actions that have consequences at user-level (of which system()
is one), but it''s not really accurate to think of them as user-level
actions;
it''s better to think of them as actions that are automatically
post-processed
in defined ways.  In system()''s case, the action indicating what
command
should be executed is itself executed in the kernel; there is no way to (for
example) act on the return value in the context of the probe that induced
the system().  This is an unfortunate limitation, but it''s a limitation
that
is a natural (and unavoidable) consequence of an arbitrary-context
instrumentation framework.  

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

On Fri, Apr 07, 2006 at 11:09:05AM -0700, Bryan Cantrill
wrote:
> > It''s how system() is implemented, IIRC.
> 
> There are actions that have consequences at user-level (of which system()
> is one), but it''s not really accurate to think of them as
user-level actions;
> it''s better to think of them as actions that are automatically
post-processed
> in defined ways.  In system()''s case, the action indicating what
command
> should be executed is itself executed in the kernel; there is no way to
(for
> example) act on the return value in the context of the probe that induced
> the system().  This is an unfortunate limitation, but it''s a
limitation that
> is a natural (and unavoidable) consequence of an arbitrary-context
> instrumentation framework.  
Right, the consumer implements the concrete action ultimately in this
case, but it''s long past the actual probe firing in the kernel, so
unless one combines such actions with stop() it''s hard to synchronously
act on the result of actions like system().

Not that one couldn''t make more use of stop()-like destructive actions
to obtain synchrony, but that it does go against the spirit of DTrace.

Nico
--

* Bryan Cantrill <bmc at eng.sun.com> [2006-04-07
09:40]:
> I think what you need is another system call interposer -- which means
> that you''ll need to architect and implement a general system call
> interposition mechanism, for which the syscall provider will
> ultimately become a client.  This would probably have value outside of
> just your work:  there are some ISVs today that develop similar
> security/auditing products that currently directly butcher the system
> call table (and thereby violate the integrity of the system in ways
> they don''t understand); it would be at least a step in the right
> direction for these ISVs to have a stable way of interposing on the
> system call table...
  If a public API is desired for those products, I would like to
  strongly contain the degree of expressiveness of such a framework;
  consider it an attempt to blunt the knives of these would-be butchers.
  As a recent example, smf(5) really expects kill(2) to deliver signals.
  On systems freshly butchered, smf(5) ends up bloodspattered and
  confused--this state is then transferred to the system admin.

  - Stephen

-- 
Stephen Hahn, PhD  Solaris Kernel Development, Sun Microsystems
stephen.hahn at sun.com  http://blogs.sun.com/sch/

Stephen Hahn wrote:
> * Bryan Cantrill <bmc at eng.sun.com> [2006-04-07 09:40]:
>> I think what you need is another system call interposer -- which means
>> that you''ll need to architect and implement a general system
call
>> interposition mechanism, for which the syscall provider will
>> ultimately become a client.  This would probably have value outside of
>> just your work:  there are some ISVs today that develop similar
>> security/auditing products that currently directly butcher the system
>> call table (and thereby violate the integrity of the system in ways
>> they don''t understand); it would be at least a step in the
right
>> direction for these ISVs to have a stable way of interposing on the
>> system call table...
> 
>   If a public API is desired for those products, I would like to
>   strongly contain the degree of expressiveness of such a framework;
>   consider it an attempt to blunt the knives of these would-be butchers.
>   As a recent example, smf(5) really expects kill(2) to deliver signals.
>   On systems freshly butchered, smf(5) ends up bloodspattered and
>   confused--this state is then transferred to the system admin.
> 
>   - Stephen
> 
In case Bryan and Stephen weren''t clear, products that attempt to
impose
a different security model on Solaris by hacking the system
call table are utterly broken.

The answer for almost all these cases is provide mechanisms
(which Solaris has done) so that users don''t need to be root
in order to perform routine administrative tasks.  Preventing
root from killing processes by hacking the system call table
in order to keep someone who should never have been given
root in the first place from killing the wrong process is
just plain wrong.

- Bart


-- 
Bart Smaalders			Solaris Kernel Performance
barts at cyber.eng.sun.com		http://blogs.sun.com/barts

On Fri 04/07/06 at 09:29 AM, bmc at eng.sun.com wrote:
> 
> you want.  Moreover, the amount of infrastructure that you''re
leveraging
> from DTrace is pretty tiny -- the system call interposition mechanism which
> you seek to reuse took just a couple of hours to implement.  I think what
> you need is another system call interposer -- which means that
you''ll need
> to architect and implement a general system call interposition mechanism,
Or use the infrastructure created by BrandZ.  Using our framework to
interpose on system calls to send a yea/nay request to a user-level server
should be fairly trivial.  This approach would constrain you to operating
inside a zone, but that might be considered a feature for a security-driven
project.

Nils

Nicolas Williams wrote:
> On Fri, Apr 07, 2006 at 07:57:06PM +0200, Casper.Dik at Sun.COM wrote:
>>> I don''t think "resistance" is quite the right
word; I think "refusal" is
>>> more the word you''re looking for.
>> I feel a OpenSolaris fork is in the air :-)
> 
> Er, not by me...  I want the feature, and though Darren and I have
> talked about building it atop DTrace I''m not wedded to that idea,
and, I
> think, neither is Darren.  Where''s the fork? :)
Indeed I certainly am not wedded to the idea of using DTrace it was more 
a way to start the discussion and find if any some commonality in the 
architecture and maybe even implementation.

I''m really glad we are having this discussion but I think it is now
time
it moved to security-discuss at opensolaris.org since we have moved away 
from the DTrace part of this discussion for now.  As such I''ve set 
Reply-To: security-disucss at opensolaris.org

-- 
Darren J Moffat