llvm dev - Nov 2015 - [RFC] A new intrinsic, `llvm.blackbox`, to explicitly prevent constprop, die, etc optimizations

On Wed, 11 Nov 2015 11:13:43 -0800, Daniel Berlin via llvm-dev wrote:
<snip for gmane>
> Heck, i could even reason about inline asm if i wanted to ;-).
> 
> My point is that this call is super special compared to all other 
> calls,
> and literally everything in LLVM has to understand that.
> The liklihood of subtle bugs being introduced in functionality (IE
> analysis/etc doing the wrong thing because it is not special cased)
> seems super high to me.
I do agree this is a concern.
 
>> I don't know how you could practically deploy a super-duper LTO
mode
>> that doesn't allow that as part of its model.
>>
>>
> Sure.
> 
> 
>> The following CFG simplification would be legal, as it also fits the
>> normal model of an external call:
>> if (cond) y =llvm.blackbox(x)
>> else y = llvm.blackbox(x)
>> -->
>> y = llvm.blackbox(x)
>>
>> I don't see how this is special. It just provides an overloaded
>> intrinsic whose definition we promise to never reason about. Other than
>> that it follows the same familiar rules that function calls do.
>>
>>
> You have now removed some conditional evaluation and  jumps.  those 
> would normally take benchmark time.
> Why is that okay?
Because the original post in terms of wanting to inhibit specific 
optimizations was a flawed way of describing the problem.

Reid's explanation of "an external function that LLVM is not allowed to
reason about the body of" is a much better explanation, as a good 
benchmark will place llvm.blackbox() exactly where real code would call, 
say, getrandom() (on input) or printf() (on output).

However, as the function call overhead of said external function isn't 
part of the _developer's_ code, and not something they can make faster in 
case of slow results, it's not relevant to the benchmarks - thus, using 
an _actual_ external function is suboptimal, even leaving aside that with 
LTO and such, llvm may STILL infer things about such functions, obviating 
the benchark.

Perhaps the best explanation is that it's about *simulating the 
existence* of a "perfectly efficient" external world.

Hi,

I don't understand why in your benchmarks you're not just using an
external
function, and then have an empty benchmark that just calls that external
function so you can know what overhead that adds. A bunch of benchmark
suites do this already, and chandler mentions it in the talk posted
earlier.

James
On Wed, 11 Nov 2015 at 21:02, Alex Elsayed via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Wed, 11 Nov 2015 11:13:43 -0800, Daniel Berlin via llvm-dev wrote:
> <snip for gmane>
> > Heck, i could even reason about inline asm if i wanted to ;-).
> >
> > My point is that this call is super special compared to all other
> > calls,
> > and literally everything in LLVM has to understand that.
> > The liklihood of subtle bugs being introduced in functionality (IE
> > analysis/etc doing the wrong thing because it is not special cased)
> > seems super high to me.
>
> I do agree this is a concern.
>
> >> I don't know how you could practically deploy a super-duper
LTO mode
> >> that doesn't allow that as part of its model.
> >>
> >>
> > Sure.
> >
> >
> >> The following CFG simplification would be legal, as it also fits
the
> >> normal model of an external call:
> >> if (cond) y =llvm.blackbox(x)
> >> else y = llvm.blackbox(x)
> >> -->
> >> y = llvm.blackbox(x)
> >>
> >> I don't see how this is special. It just provides an
overloaded
> >> intrinsic whose definition we promise to never reason about. Other
than
> >> that it follows the same familiar rules that function calls do.
> >>
> >>
> > You have now removed some conditional evaluation and  jumps.  those
> > would normally take benchmark time.
> > Why is that okay?
>
> Because the original post in terms of wanting to inhibit specific
> optimizations was a flawed way of describing the problem.
>
> Reid's explanation of "an external function that LLVM is not
allowed to
> reason about the body of" is a much better explanation, as a good
> benchmark will place llvm.blackbox() exactly where real code would call,
> say, getrandom() (on input) or printf() (on output).
>
> However, as the function call overhead of said external function isn't
> part of the _developer's_ code, and not something they can make faster
in
> case of slow results, it's not relevant to the benchmarks - thus, using
> an _actual_ external function is suboptimal, even leaving aside that with
> LTO and such, llvm may STILL infer things about such functions, obviating
> the benchark.
>
> Perhaps the best explanation is that it's about *simulating the
> existence* of a "perfectly efficient" external world.
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151111/98bee232/attachment.html>

>
>
>
> Reid's explanation of "an external function that LLVM is not
allowed to
> reason about the body of" is a much better explanation, as a good
> benchmark will place llvm.blackbox() exactly where real code would call,
> say, getrandom() (on input) or printf() (on output).
>
> However, as the function call overhead of said external function isn't
> part of the _developer's_ code,

This isn't call overhead though.
It's a conditional and two calls someone wrote in some benchmark code.
That's not call overhead ;-)

It's just that i've proven the condition has no side effects and
doesn't
matter, so i eliminated it.

Thus, I'm trying to ask the question: "Will the use case really still
be
served if we let us eliminate these conditionals as useless, when the whole
point is to let people test the overhead of things the compiler wanted to
eliminate because it thinks they are useless"
;-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151111/64cb38b4/attachment.html>

On Wed, 11 Nov 2015 13:15:45 -0800, Daniel Berlin via llvm-dev wrote:

>>
>>
>> Reid's explanation of "an external function that LLVM is not
allowed to
>> reason about the body of" is a much better explanation, as a good
>> benchmark will place llvm.blackbox() exactly where real code would
>> call,
>> say, getrandom() (on input) or printf() (on output).
>>
>>
> 
>> However, as the function call overhead of said external function
isn't
>> part of the _developer's_ code,
> 
> 
> This isn't call overhead though.
> It's a conditional and two calls someone wrote in some benchmark code.
> That's not call overhead ;-)
I meant the prologue/epilogue of the external function, but James' 
response is relevant there.
> It's just that i've proven the condition has no side effects and
doesn't
> matter, so i eliminated it.
Yes. That's perfectly fine. You could do the exact same thing with 
getrandom()'s result, or printf() calls.

That was my point.
> Thus, I'm trying to ask the question: "Will the use case really
still be
> served if we let us eliminate these conditionals as useless, when the
> whole point is to let people test the overhead of things the compiler
> wanted to eliminate because it thinks they are useless"
> ;-)
And my answer was "Yes, emphatically so, as you're continually
restating
what I consider a deeply flawed summation of what it's trying to
solve."
> the whole point is to let people test the overhead of things the
> compiler wanted to eliminate because it thinks they are useless"
This is _incorrect_.

The point is to _model the behavior of the benchmarked code AS IF the 
data goes to and comes from a place we know nothing about_.

These are fundamentally different things, which is why I keep restating 
it.

On Wed, Nov 11, 2015 at 1:01 PM, Alex Elsayed via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Wed, 11 Nov 2015 11:13:43 -0800, Daniel Berlin via llvm-dev wrote:
> <snip for gmane>
> > Heck, i could even reason about inline asm if i wanted to ;-).
> >
> > My point is that this call is super special compared to all other
> > calls,
> > and literally everything in LLVM has to understand that.
> > The liklihood of subtle bugs being introduced in functionality (IE
> > analysis/etc doing the wrong thing because it is not special cased)
> > seems super high to me.
>
> I do agree this is a concern.
>
> >> I don't know how you could practically deploy a super-duper
LTO mode
> >> that doesn't allow that as part of its model.
> >>
> >>
> > Sure.
> >
> >
> >> The following CFG simplification would be legal, as it also fits
the
> >> normal model of an external call:
> >> if (cond) y =llvm.blackbox(x)
> >> else y = llvm.blackbox(x)
> >> -->
> >> y = llvm.blackbox(x)
> >>
> >> I don't see how this is special. It just provides an
overloaded
> >> intrinsic whose definition we promise to never reason about. Other
than
> >> that it follows the same familiar rules that function calls do.
> >>
> >>
> > You have now removed some conditional evaluation and  jumps.  those
> > would normally take benchmark time.
> > Why is that okay?
>
> Because the original post in terms of wanting to inhibit specific
> optimizations was a flawed way of describing the problem.
>
> Reid's explanation of "an external function that LLVM is not
allowed to
> reason about the body of" is a much better explanation, as a good
> benchmark will place llvm.blackbox() exactly where real code would call,
> say, getrandom() (on input) or printf() (on output).
>
> However, as the function call overhead of said external function isn't
> part of the _developer's_ code, and not something they can make faster
in
> case of slow results, it's not relevant to the benchmarks - thus, using
> an _actual_ external function is suboptimal, even leaving aside that with
> LTO and such, llvm may STILL infer things about such functions, obviating
> the benchark.
>
> Perhaps the best explanation is that it's about *simulating the
> existence* of a "perfectly efficient" external world.
>
Can you show a real benchmark that users have tried to write where the call
overhead of actually using an external function call is measurable? A no-op
function call is going to take maybe a dozen cycles max (inside a loop, so
good branch prediction etc.). Anything where a dozen cycles is measurable
by comparison basically can't be reasoned about at the C++ level (you are
basically benchmarking at asm level at that point, so just write it in asm).

More generally, there's only been (I think) 1 concrete example given in
this thread (the xor fold thing). Could you please give like 5 distinct
real-world examples? That would help us get a feel for the real motivation
here and why an external function call wouldn't work. (also, presumably
this is a consistent problem that has been cropping up in practice if you
are going to the length of wanting to add an IR intrinsic that, as Daniel
points out, has implications throughout the compiler)

-- Sean Silva

>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151111/bc66a276/attachment.html>

On Wed, Nov 11, 2015 at 07:14:28PM -0800, Sean Silva via llvm-dev
wrote:
> Can you show a real benchmark that users have tried to write where the call
> overhead of actually using an external function call is measurable?
This is the wrong question. The correct question is: What useful
benchmark cannot trivally factor out the overhead of the external
function call. Yes, if you do microbenchmarking it can be measurable.
But the point is that the overhead should be extremely predictable and
stable. As such, it can be easily calibrated and removed from the cost
of whatever you are really trying to measure. Given that the
instrumentation in general has some latency, you won't get around
calibration anyway.

Joerg