Linux Virtualization - Nov 2013 - [PATCH -tip v3 00/23] kprobes: introduce NOKPROBE_SYMBOL() and general cleaning of kprobe blacklist

Hi -
> > Does this new blacklist cover enough that the kernel now survives a 
> > broadly wildcarded perf-probe, e.g. over e.g. all of its kallsyms?
> 
> That's generally the purpose of the annotations - if it doesn't
then
> that's a bug.
AFAIK, no kernel since kprobes was introduced has ever stood up to
that test.  perf probe lacks the wildcarding powers of systemtap, so
one needs to resort to something like:

# cat /proc/kallsyms | grep ' [tT] ' | while read addr type symbol; do
   perf probe $symbol
done

then wait for a few hours for that to finish. Then, or while the loop
is still running, run

# perf record -e 'probe:*' -aR sleep 1

to take a kernel down.


- FChE

On Wed, 20 Nov 2013 12:36:00 -0500
"Frank Ch. Eigler" <fche at redhat.com> wrote:
> Hi -
> 
> > > Does this new blacklist cover enough that the kernel now survives
a
> > > broadly wildcarded perf-probe, e.g. over e.g. all of its
kallsyms?
> > 
> > That's generally the purpose of the annotations - if it
doesn't then
> > that's a bug.
> 
> AFAIK, no kernel since kprobes was introduced has ever stood up to
> that test.  perf probe lacks the wildcarding powers of systemtap, so
> one needs to resort to something like:
> 
> # cat /proc/kallsyms | grep ' [tT] ' | while read addr type symbol;
do
>    perf probe $symbol
> done
I'm curious to why one would do that. IIUC, perf now has function
tracing support.

-- Steve
> 
> then wait for a few hours for that to finish. Then, or while the loop
> is still running, run
> 
> # perf record -e 'probe:*' -aR sleep 1
> 
> to take a kernel down.
> 
> 
> - FChE

On 11/20/2013 09:56 AM, Steven Rostedt wrote:
> On Wed, 20 Nov 2013 12:36:00 -0500
> "Frank Ch. Eigler" <fche at redhat.com> wrote:
> 
>> Hi -
>>
>>>> Does this new blacklist cover enough that the kernel now
survives a
>>>> broadly wildcarded perf-probe, e.g. over e.g. all of its
kallsyms?
>>>
>>> That's generally the purpose of the annotations - if it
doesn't then
>>> that's a bug.
>>
>> AFAIK, no kernel since kprobes was introduced has ever stood up to
>> that test.  perf probe lacks the wildcarding powers of systemtap, so
>> one needs to resort to something like:
>>
>> # cat /proc/kallsyms | grep ' [tT] ' | while read addr type
symbol; do
>>    perf probe $symbol
>> done
> 
> I'm curious to why one would do that. IIUC, perf now has function
> tracing support.
Then consider something like probing all inline "call" sites, which
will
be sprinkled in the middle where ftrace doesn't apply.

The point is not whether there's an alternative - kprobes really ought
to be wholly safe regardless.  Slow, if you did such broad probing,
sure, but still safe.

And a real use-case probably wouldn't probe *all* functions/inlines, but
it illustrates that there are at least a few in the full set that don't
behave well.

(2013/11/21 2:36), Frank Ch. Eigler wrote:
> Hi -
> 
>>> Does this new blacklist cover enough that the kernel now survives a
>>> broadly wildcarded perf-probe, e.g. over e.g. all of its kallsyms?
>>
>> That's generally the purpose of the annotations - if it doesn't
then
>> that's a bug.
> 
> AFAIK, no kernel since kprobes was introduced has ever stood up to
> that test.  perf probe lacks the wildcarding powers of systemtap, so
> one needs to resort to something like:
> 
> # cat /proc/kallsyms | grep ' [tT] ' | while read addr type symbol;
do
>    perf probe $symbol
> done
> 
> then wait for a few hours for that to finish. Then, or while the loop
> is still running, run
> 
> # perf record -e 'probe:*' -aR sleep 1
> 
> to take a kernel down.
Um, indeed, current blacklist is not perfect. As I reported in this
series, I've found 2 more patterns. I guess there still have some
others.

But anyway, I don't think it is good to fix all such bugs
in this series.
This is just the first step to do it. :)

Thank you,

-- 
Masami HIRAMATSU
IT Management Research Dept. Linux Technology Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt at hitachi.com

* Masami Hiramatsu <masami.hiramatsu.pt at hitachi.com> wrote:
> (2013/11/21 2:36), Frank Ch. Eigler wrote:
[ ... ]
> > one needs to resort to something like:
> > 
> > # cat /proc/kallsyms | grep ' [tT] ' | while read addr type
symbol; do
> >    perf probe $symbol
> > done
> > 
> > then wait for a few hours for that to finish. Then, or while the loop
> > is still running, run
> > 
> > # perf record -e 'probe:*' -aR sleep 1
> > 
> > to take a kernel down.
> 
> Um, indeed, current blacklist is not perfect. [...]
Then it needs to be fixed ASAP!
> [...] As I reported in this series, I've found 2 more patterns. I 
> guess there still have some others.
> 
> But anyway, I don't think it is good to fix all such bugs in this 
> series.
Fixing these bugs is far more important than any cleanup work.

We can apply the fixes together with your cleanup series to make it 
all simpler, but the bug fixing absolutely needs to happen right now.

Thanks,

	Ingo