Virtualization - Nov 2013 - [PATCH -tip v3 00/23] kprobes: introduce NOKPROBE_SYMBOL() and general cleaning of kprobe blacklist

(2013/11/22 11:35), Masami Hiramatsu wrote:
> (2013/11/21 16:29), Ingo Molnar wrote:
>>
>> * Masami Hiramatsu <masami.hiramatsu.pt at hitachi.com> wrote:
>>
>>> (2013/11/21 2:36), Frank Ch. Eigler wrote:
>>
>> [ ... ]
>>>> one needs to resort to something like:
>>>>
>>>> # cat /proc/kallsyms | grep ' [tT] ' | while read addr
type symbol; do
>>>>    perf probe $symbol
>>>> done
>>>>
>>>> then wait for a few hours for that to finish. Then, or while
the loop
>>>> is still running, run
>>>>
>>>> # perf record -e 'probe:*' -aR sleep 1
>>>>
>>>> to take a kernel down.
>>>
>>> Um, indeed, current blacklist is not perfect. [...]
>>
>> Then it needs to be fixed ASAP!
> 
> OK, I see. At least the two patches included this series
> should be fixed. :)
> 
> And more, I need to test all symbols and drills down.
OK, what I've found was;
 - The functions which can be ftraced look good.
   (see tracing/available_filter_functions)
 - following functions should not be able to be probed.
   - memcpy, memset
   - native_load_sp0 and some other native functions (need to be clear)
   - restore
   - trace_graph_return
   - trace_hardirqs_off_thunk, trace_hardirqs_on_thunk
   - This list still be not perfect. I just enabled/disabled kprobes
     one by one. There might be combined bugs (combination of several
     kprobes).
 - Some of them are hard to specify by NOKPROBE_SYMBOL because they are
   defined in assembly file.

Anyway, to fix all of them, I think we need file-based blacklist
especially for assembler symbols.

For example, we can get all text symbols by below command;
 nm some-file.o | grep -i " t " | cut -f3 -d" "
so that we can make a blacklisted-symbol list for the file.
I need to look the Kbuild for how I can do that in Makefile.

Thank you,

-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt at hitachi.com

* Masami Hiramatsu <masami.hiramatsu.pt at hitachi.com> wrote:
> (2013/11/22 11:35), Masami Hiramatsu wrote:
> > (2013/11/21 16:29), Ingo Molnar wrote:
> >>
> >> * Masami Hiramatsu <masami.hiramatsu.pt at hitachi.com>
wrote:
> >>
> >>> (2013/11/21 2:36), Frank Ch. Eigler wrote:
> >>
> >> [ ... ]
> >>>> one needs to resort to something like:
> >>>>
> >>>> # cat /proc/kallsyms | grep ' [tT] ' | while read
addr type symbol; do
> >>>>    perf probe $symbol
> >>>> done
> >>>>
> >>>> then wait for a few hours for that to finish. Then, or
while the loop
> >>>> is still running, run
> >>>>
> >>>> # perf record -e 'probe:*' -aR sleep 1
> >>>>
> >>>> to take a kernel down.
> >>>
> >>> Um, indeed, current blacklist is not perfect. [...]
> >>
> >> Then it needs to be fixed ASAP!
> > 
> > OK, I see. At least the two patches included this series
> > should be fixed. :)
> > 
> > And more, I need to test all symbols and drills down.
> 
> OK, what I've found was;
>  - The functions which can be ftraced look good.
>    (see tracing/available_filter_functions)
>  - following functions should not be able to be probed.
>    - memcpy, memset
>    - native_load_sp0 and some other native functions (need to be clear)
>    - restore
>    - trace_graph_return
>    - trace_hardirqs_off_thunk, trace_hardirqs_on_thunk
>    - This list still be not perfect. I just enabled/disabled kprobes
>      one by one. There might be combined bugs (combination of several
>      kprobes).
>  - Some of them are hard to specify by NOKPROBE_SYMBOL because they are
>    defined in assembly file.
> 
> Anyway, to fix all of them, I think we need file-based blacklist
> especially for assembler symbols.
assembler symbols shouldn't be particular hard either, just put them 
into the noprobes section.
> For example, we can get all text symbols by below command;
>  nm some-file.o | grep -i " t " | cut -f3 -d" "
> so that we can make a blacklisted-symbol list for the file.
> I need to look the Kbuild for how I can do that in Makefile.
I think it's generally better to add explicit annotations to the code.

Thanks,

	Ingo

(2013/11/27 22:30), Ingo Molnar wrote:
> 
> * Masami Hiramatsu <masami.hiramatsu.pt at hitachi.com> wrote:
> 
>> (2013/11/22 11:35), Masami Hiramatsu wrote:
>>> (2013/11/21 16:29), Ingo Molnar wrote:
>>>>
>>>> * Masami Hiramatsu <masami.hiramatsu.pt at hitachi.com>
wrote:
>>>>
>>>>> (2013/11/21 2:36), Frank Ch. Eigler wrote:
>>>>
>>>> [ ... ]
>>>>>> one needs to resort to something like:
>>>>>>
>>>>>> # cat /proc/kallsyms | grep ' [tT] ' | while
read addr type symbol; do
>>>>>>    perf probe $symbol
>>>>>> done
>>>>>>
>>>>>> then wait for a few hours for that to finish. Then, or
while the loop
>>>>>> is still running, run
>>>>>>
>>>>>> # perf record -e 'probe:*' -aR sleep 1
>>>>>>
>>>>>> to take a kernel down.
>>>>>
>>>>> Um, indeed, current blacklist is not perfect. [...]
>>>>
>>>> Then it needs to be fixed ASAP!
>>>
>>> OK, I see. At least the two patches included this series
>>> should be fixed. :)
>>>
>>> And more, I need to test all symbols and drills down.
>>
>> OK, what I've found was;
>>  - The functions which can be ftraced look good.
>>    (see tracing/available_filter_functions)
>>  - following functions should not be able to be probed.
>>    - memcpy, memset
>>    - native_load_sp0 and some other native functions (need to be clear)
>>    - restore
>>    - trace_graph_return
>>    - trace_hardirqs_off_thunk, trace_hardirqs_on_thunk
>>    - This list still be not perfect. I just enabled/disabled kprobes
>>      one by one. There might be combined bugs (combination of several
>>      kprobes).
>>  - Some of them are hard to specify by NOKPROBE_SYMBOL because they are
>>    defined in assembly file.
>>
>> Anyway, to fix all of them, I think we need file-based blacklist
>> especially for assembler symbols.
> 
> assembler symbols shouldn't be particular hard either, just put them 
> into the noprobes section.
Would you mean .kprobes.text? Hmm, I hope not to use it anymore,
but yeah, bugfix is more important. Agreed.
>> For example, we can get all text symbols by below command;
>>  nm some-file.o | grep -i " t " | cut -f3 -d" "
>> so that we can make a blacklisted-symbol list for the file.
>> I need to look the Kbuild for how I can do that in Makefile.
> 
> I think it's generally better to add explicit annotations to the code.
OK, thus I'll add annotations first.

Thank you!

-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt at hitachi.com