Sean Silva via llvm-dev
2015-Nov-17 04:55 UTC
[llvm-dev] [RFC] A new intrinsic, `llvm.blackbox`, to explicitly prevent constprop, die, etc optimizations
On Mon, Nov 16, 2015 at 6:59 PM, Dmitri Gribenko via llvm-dev < llvm-dev at lists.llvm.org> wrote:> On Mon, Nov 16, 2015 at 10:03 AM, James Molloy via llvm-dev > <llvm-dev at lists.llvm.org> wrote: > > You don't appear to have addressed my suggestion to not require a perfect > > external world, instead to measure the overhead of an imperfect world (by > > using an empty benchmark) and subtracting that from the measured > benchmark > > score. > > In microbenchmarks, performance is not additive. You can't compose > two pieces of code and predict that the benchmark results will be the > sum of the individual measurements. >This sounds like an argument against microbenchmarks in general, rather than against James's point. James' point assumes that you are doing a meaningful benchmark (since, presumably, you are trying to understand the relative performance in your application). -- Sean Silva> > Dmitri > > -- > main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if > (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at gmail.com>*/ > _______________________________________________ > LLVM Developers mailing list > llvm-dev at lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151116/e111c83c/attachment.html>
Dmitri Gribenko via llvm-dev
2015-Nov-17 05:07 UTC
[llvm-dev] [RFC] A new intrinsic, `llvm.blackbox`, to explicitly prevent constprop, die, etc optimizations
On Mon, Nov 16, 2015 at 8:55 PM, Sean Silva <chisophugis at gmail.com> wrote:> > > On Mon, Nov 16, 2015 at 6:59 PM, Dmitri Gribenko via llvm-dev > <llvm-dev at lists.llvm.org> wrote: >> >> On Mon, Nov 16, 2015 at 10:03 AM, James Molloy via llvm-dev >> <llvm-dev at lists.llvm.org> wrote: >> > You don't appear to have addressed my suggestion to not require a >> > perfect >> > external world, instead to measure the overhead of an imperfect world >> > (by >> > using an empty benchmark) and subtracting that from the measured >> > benchmark >> > score. >> >> In microbenchmarks, performance is not additive. You can't compose >> two pieces of code and predict that the benchmark results will be the >> sum of the individual measurements. > > > This sounds like an argument against microbenchmarks in general, rather than > against James's point. James' point assumes that you are doing a meaningful > benchmark (since, presumably, you are trying to understand the relative > performance in your application).Any benchmarking, and especially microbenchmarking, should not be primarily about measuring the relative performance change. It is a small scientific experiment, where you don't just get numbers -- you need to have an explanation why are you getting these numbers. And, especially for microbenchmarks, having an explanation, and a way to validate it, as well as one's assumptions, is critical. In large system benchmarks performance is not additive either -- when you have multiple subsystems, cores and queues. But this does not mean that system-level benchmarks are not useful. As any benchmarks, they need interpretation. Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at gmail.com>*/
Sean Silva via llvm-dev
2015-Nov-17 05:34 UTC
[llvm-dev] [RFC] A new intrinsic, `llvm.blackbox`, to explicitly prevent constprop, die, etc optimizations
On Mon, Nov 16, 2015 at 9:07 PM, Dmitri Gribenko <gribozavr at gmail.com> wrote:> On Mon, Nov 16, 2015 at 8:55 PM, Sean Silva <chisophugis at gmail.com> wrote: > > > > > > On Mon, Nov 16, 2015 at 6:59 PM, Dmitri Gribenko via llvm-dev > > <llvm-dev at lists.llvm.org> wrote: > >> > >> On Mon, Nov 16, 2015 at 10:03 AM, James Molloy via llvm-dev > >> <llvm-dev at lists.llvm.org> wrote: > >> > You don't appear to have addressed my suggestion to not require a > >> > perfect > >> > external world, instead to measure the overhead of an imperfect world > >> > (by > >> > using an empty benchmark) and subtracting that from the measured > >> > benchmark > >> > score. > >> > >> In microbenchmarks, performance is not additive. You can't compose > >> two pieces of code and predict that the benchmark results will be the > >> sum of the individual measurements. > > > > > > This sounds like an argument against microbenchmarks in general, rather > than > > against James's point. James' point assumes that you are doing a > meaningful > > benchmark (since, presumably, you are trying to understand the relative > > performance in your application). > > Any benchmarking, and especially microbenchmarking, should not be > primarily about measuring the relative performance change. It is a > small scientific experiment, where you don't just get numbers -- you > need to have an explanation why are you getting these numbers. And, > especially for microbenchmarks, having an explanation, and a way to > validate it, as well as one's assumptions, is critical. > > In large system benchmarks performance is not additive either -- when > you have multiple subsystems, cores and queues. But this does not > mean that system-level benchmarks are not useful. As any benchmarks, > they need interpretation. >I agree with all this, but I don't understand how adding an external function call would interfere at all. I guess I could rephrase what I was saying as "if having an external function call prevents the benchmark from performing its purpose, it is hard to believe that any conclusions coming from such a benchmark would be applicable to real code". The only exceptions I can think about are extremely low-level asm measurements (which are written in asm anyway so this whole discussion of llvm.blackbox is irrelevant). -- Sean Silva> > Dmitri > > -- > main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if > (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at gmail.com>*/ >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20151116/4e37ec0a/attachment.html>