dtrace discuss - May 2009 - can Dtrace be used for the error injection?

Hi,

is it somehow possible to use Dtrace for error injection in a kernel module?

Something like changing:
- function return value
-  value of a register

If not, can it be implemented?


I  can do that via kmdb, but I need Dtrace for the time synchronization 
- chill() action.
I can not combine Dtrace & kmdb:

 dtrace: failed to initialize dtrace: DTrace cannot be used when kernel 
debugger is active

Thanks,
Pavel

On Sat, May 30, 2009 at 12:04:50AM +0200, Pavel Filipensky
wrote:
> Hi,
> 
> is it somehow possible to use Dtrace for error injection in a kernel
module?
> 
> Something like changing:
> - function return value
> -  value of a register
> 
> If not, can it be implemented?
No;  it violates the basic safety constraints of dtrace(1M).
> I  can do that via kmdb, but I need Dtrace for the time synchronization 
> - chill() action.
> I can not combine Dtrace & kmdb:
> 
> dtrace: failed to initialize dtrace: DTrace cannot be used when kernel 
> debugger is active
In kmdb:

...]> kdi_dtrace_state/W 1

will allow dtrace to start.  If it breaks, you get to keep both pieces.

If you put a debugger breakpoint at the same place a dtrace tracepoint is,
things will not be happy.

Cheers,
- jonathan

Hello Jonathan,

Saturday, May 30, 2009, 12:11:58 AM, you wrote:

JA> On Sat, May 30, 2009 at 12:04:50AM +0200, Pavel Filipensky
wrote:
>> Hi,
>> 
>> is it somehow possible to use Dtrace for error injection in a kernel
module?
>> 
>> Something like changing:
>> - function return value
>> -  value of a register
>> 
>> If not, can it be implemented?
JA> No;  it violates the basic safety constraints of dtrace(1M).

I don''t know... it allows to change some things like n structure in
user space which is filled in in kernel by syscall, but then doesn''t
allow to change an return code. What I''m trying to say is that it
already allows to shoot yourself in foot or fix rather help you (like
changing uname output being a classic example now) but doesn''t allow
you to do so with other cases...

I think it would be very useful if it wuld allow to assign values to
argN (args[N}).

-- 
Best regards,
 Robert Milkowski
                                       http://milek.blogspot.com

It''s easier to create in interposer library which will look somewhat
like this:

/*
 * pthread_mutex_lock -- intercept pthread_mutex_lock
 */
int pthread_mutex_lock(pthread_mutex_t *mutex){
        static void *actual_pthread_mutex_lock = NULL;

        if (actual_pthread_mutex_lock == NULL) {
                actual_pthread_mutex_lock = dlsym(RTLD_NEXT,
			 "pthread_mutex_lock");
                assert(actual_pthread_mutex_lock != NULL);
        }
        return ((int (*)(pthread_mutex_t *))
		actual_pthread_mutex_lock)(mutex);
}

This is nominally limited to user-space, but can happily
generate all the errors you''d ever want (;-))

You can do the same thing for kernel modules, by inserting
a module with the error injection code in front of a
hacked regular module.

--dave




Robert Milkowski <milek at task.gda.pl> writes
> Hello Jonathan,
> 
> Saturday, May 30, 2009, 12:11:58 AM, you wrote:
> 
> JA> On Sat, May 30, 2009 at 12:04:50AM +0200, Pavel Filipensky wrote:
>>> Hi,
>>>
>>> is it somehow possible to use Dtrace for error injection in a
kernel module?
>>>
>>> Something like changing:
>>> - function return value
>>> -  value of a register
>>>
>>> If not, can it be implemented?
> 
> JA> No;  it violates the basic safety constraints of dtrace(1M).
> 
> I don''t know... it allows to change some things like n structure
in
> user space which is filled in in kernel by syscall, but then
doesn''t
> allow to change an return code. What I''m trying to say is that it
> already allows to shoot yourself in foot or fix rather help you (like
> changing uname output being a classic example now) but doesn''t
allow
> you to do so with other cases...
> 
> I think it would be very useful if it wuld allow to assign values to
> argN (args[N}).
> 
-- 
David Collier-Brown            | Always do right. This will gratify
Sun Microsystems, Toronto      | some people and astonish the rest
davecb at sun.com                 |                      -- Mark Twain
cell: (647) 833-9377, home (416) 223-8968, bridge (877) 385-4099 code
506 9191#

On Sun, May 31, 2009 at 06:47:55PM +0100, Robert Milkowski
wrote:
> Hello Jonathan,
> 
> Saturday, May 30, 2009, 12:11:58 AM, you wrote:
> 
> JA> On Sat, May 30, 2009 at 12:04:50AM +0200, Pavel Filipensky wrote:
> >> Hi,
> >> 
> >> is it somehow possible to use Dtrace for error injection in a
kernel module?
> >> 
> >> Something like changing:
> >> - function return value
> >> -  value of a register
> >> 
> >> If not, can it be implemented?
> 
> JA> No;  it violates the basic safety constraints of dtrace(1M).
> 
> I don''t know... it allows to change some things like n structure
in
> user space which is filled in in kernel by syscall, but then
doesn''t
> allow to change an return code. What I''m trying to say is that it
> already allows to shoot yourself in foot or fix rather help you (like
> changing uname output being a classic example now) but doesn''t
allow
> you to do so with other cases...
But none of those should let you crash the system, since arguments from
the user are checked for validity before the kernel uses them.  You can destroy
user process state, but not kernel state.

If you can change kernel argument values in arbitrary ways, there''s no
way
to guarantee the correctness of the result.

We have talked about DTRACE_PROBE() calls which return a result, which
would allow targetted error injection.  But any such call will require
kernel modification to work;  you''re not ever going to be able to have
kmem_alloc(nnn, KM_SLEEP) return NULL.
> I think it would be very useful if it wuld allow to assign values to
> argN (args[N}).
For syscall and pid provider probes, that''s possible.  For kernel-level
probes,
I''d say never.

Cheers,
- jonathan