llvm dev - Nov 2015 - [RFC] A new intrinsic, `llvm.blackbox`, to explicitly prevent constprop, die, etc optimizations

On Mon, Nov 16, 2015 at 6:59 PM, Dmitri Gribenko via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
> On Mon, Nov 16, 2015 at 10:03 AM, James Molloy via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > You don't appear to have addressed my suggestion to not require a
perfect
> > external world, instead to measure the overhead of an imperfect world
(by
> > using an empty benchmark) and subtracting that from the measured
> benchmark
> > score.
>
> In microbenchmarks, performance is not additive.  You can't compose
> two pieces of code and predict that the benchmark results will be the
> sum of the individual measurements.
>
This sounds like an argument against microbenchmarks in general, rather
than against James's point. James' point assumes that you are doing a
meaningful benchmark (since, presumably, you are trying to understand the
relative performance in your application).

-- Sean Silva

>
> Dmitri
>
> --
> main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
> (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at
gmail.com>*/
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151116/e111c83c/attachment.html>

On Mon, Nov 16, 2015 at 8:55 PM, Sean Silva <chisophugis at gmail.com>
wrote:
>
>
> On Mon, Nov 16, 2015 at 6:59 PM, Dmitri Gribenko via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>>
>> On Mon, Nov 16, 2015 at 10:03 AM, James Molloy via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>> > You don't appear to have addressed my suggestion to not
require a
>> > perfect
>> > external world, instead to measure the overhead of an imperfect
world
>> > (by
>> > using an empty benchmark) and subtracting that from the measured
>> > benchmark
>> > score.
>>
>> In microbenchmarks, performance is not additive.  You can't compose
>> two pieces of code and predict that the benchmark results will be the
>> sum of the individual measurements.
>
>
> This sounds like an argument against microbenchmarks in general, rather
than
> against James's point. James' point assumes that you are doing a
meaningful
> benchmark (since, presumably, you are trying to understand the relative
> performance in your application).
Any benchmarking, and especially microbenchmarking, should not be
primarily about measuring the relative performance change.  It is a
small scientific experiment, where you don't just get numbers -- you
need to have an explanation why are you getting these numbers.  And,
especially for microbenchmarks, having an explanation, and a way to
validate it, as well as one's assumptions, is critical.

In large system benchmarks performance is not additive either -- when
you have multiple subsystems, cores and queues.  But this does not
mean that system-level benchmarks are not useful.  As any benchmarks,
they need interpretation.

Dmitri

-- 
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at
gmail.com>*/

On Mon, Nov 16, 2015 at 9:07 PM, Dmitri Gribenko <gribozavr at gmail.com>
wrote:
> On Mon, Nov 16, 2015 at 8:55 PM, Sean Silva <chisophugis at
gmail.com> wrote:
> >
> >
> > On Mon, Nov 16, 2015 at 6:59 PM, Dmitri Gribenko via llvm-dev
> > <llvm-dev at lists.llvm.org> wrote:
> >>
> >> On Mon, Nov 16, 2015 at 10:03 AM, James Molloy via llvm-dev
> >> <llvm-dev at lists.llvm.org> wrote:
> >> > You don't appear to have addressed my suggestion to not
require a
> >> > perfect
> >> > external world, instead to measure the overhead of an
imperfect world
> >> > (by
> >> > using an empty benchmark) and subtracting that from the
measured
> >> > benchmark
> >> > score.
> >>
> >> In microbenchmarks, performance is not additive.  You can't
compose
> >> two pieces of code and predict that the benchmark results will be
the
> >> sum of the individual measurements.
> >
> >
> > This sounds like an argument against microbenchmarks in general,
rather
> than
> > against James's point. James' point assumes that you are doing
a
> meaningful
> > benchmark (since, presumably, you are trying to understand the
relative
> > performance in your application).
>
> Any benchmarking, and especially microbenchmarking, should not be
> primarily about measuring the relative performance change.  It is a
> small scientific experiment, where you don't just get numbers -- you
> need to have an explanation why are you getting these numbers.  And,
> especially for microbenchmarks, having an explanation, and a way to
> validate it, as well as one's assumptions, is critical.
>
> In large system benchmarks performance is not additive either -- when
> you have multiple subsystems, cores and queues.  But this does not
> mean that system-level benchmarks are not useful.  As any benchmarks,
> they need interpretation.
>
I agree with all this, but I don't understand how adding an external
function call would interfere at all. I guess I could rephrase what I was
saying as "if having an external function call prevents the benchmark from
performing its purpose, it is hard to believe that any conclusions coming
from such a benchmark would be applicable to real code". The only
exceptions I can think about are extremely low-level asm measurements
(which are written in asm anyway so this whole discussion of llvm.blackbox
is irrelevant).

-- Sean Silva

>
> Dmitri
>
> --
> main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
> (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at
gmail.com>*/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.llvm.org/pipermail/llvm-dev/attachments/20151116/4e37ec0a/attachment.html>