search for: 10&plot

I've set up a public LNT server to show the result of perf stat. There is a huge improvement compared with timeit tool. http://parkas16.inria.fr:8000/ Patch is updated to pin the process to a single core, the readings are even more accurate. It's hard coded to run everything on core 0, so don't run parallel testing with it for now. The tool now depends on Linux perf and schedtool.

...er, etc. >> > > Interesting. In fact I had the very same thoughts at the beginning. > > However, when looking at my test results the common pattern looks like > this example: > > http://llvm.org/perf/db_default/v4/nts/graph?show_all_ > points=yes&moving_window_size=10&plot.0=34.95.3&submit=Update > > The run-time of a test case is very consistently one of several fixed > values. The distribution of the different times is very consistent and > seems to form, in fact, something like a normal distribution (more in the > center, less at the bo...

...ooked at, picking mean largely reduces the > run-over-run changes reported due to noise. > > > > See this run e.g: > > > > If we use the median, we just get just one change reported: > > > > > http://llvm.org/perf/db_default/v4/nts/20661?num_comparison_runs=10&test_filter=&test_min_value_filter=&aggregation_fn=median&compare_to=20659&submit=Update > > > > If you use min, we get eight reports one claiming over 100% performance > > reduction for a case that is really just pure noise. I am planning to > look into usi...

Thanks a lot for setting this up! By using the "mean as aggregation" option one can see the noise in the results better: http://lnt.llvm.org/db_default/v4/link/graph?switch_min_mean=yes&moving_window_size=10&plot.9=1.9.7&submit=Update There are a few benchmarknig tips in https://www.llvm.org/docs/Benchmarking.html. For example, from looking at http://lab.llvm.org:8011/builders/lld-perf-testsuite/builds/285/steps/cmake-configure/logs/stdio It seems the produced lld binary is not being static...

...t; rafael.espindola at gmail.com> wrote: > Thanks a lot for setting this up! > > By using the "mean as aggregation" option one can see the noise in the > results better: > > http://lnt.llvm.org/db_default/v4/link/graph?switch_ > min_mean=yes&moving_window_size=10&plot.9=1.9.7&submit=Update > > There are a few benchmarknig tips in https://www.llvm.org/docs/ > Benchmarking.html. > > For example, from looking at > > http://lab.llvm.org:8011/builders/lld-perf-testsuite/ > builds/285/steps/cmake-configure/logs/stdio > > It s...

...had the very same thoughts at the beginning. >>> >>> However, when looking at my test results the common pattern looks like >>> this example: >>> >>> http://llvm.org/perf/db_default/v4/nts/graph?show_all_ >>> points=yes&moving_window_size=10&plot.0=34.95.3&submit=Update >>> >>> The run-time of a test case is very consistently one of several fixed >>> values. The distribution of the different times is very consistent and >>> seems to form, in fact, something like a normal distribution (more i...

..., and so I think that a better way of phrasing it is that we should construct a list of programs in the test suite that are not useful for benchmarking. My proposed exclusion list is provided below. I constructed this exclusion list primarily based on the following experiment: I ran the test suite 10 times in three configurations: 1) On an IBM POWER7 (P7) with -O3 -mvsx, 2) On a P7 at -O0 and 3) On an Intel Xeon E5430 with -O3 all using make -j6. I then used the ministat utility (which performs a T test) to compare the timings of the two P7 configurations against each other and the Xeon configu...

...machine noise is expected. Median is more tolerant to machine noise, therefore they disappear. As suggested by Chandler, we should also lock the CPU frequency to further reduce machine noise. > > http://parkas16.inria.fr:8000/db_default/v4/nts/graph?show_all_points=yes&moving_window_size=10&plot.0=1.428.3&submit=Update > > Cheers, > Tobias

Hello George, Sorry, somehow hit a send button too soon. Please ignore the previous e-mail. The bot does 10 runs for each of the benchmarks (those dots in the logs are meaningful). We can increase the number of runs if proven that this would significantly increase the accuracy. I didn't see the increase in accuracy when have been staging the bot, which would justify the extra time and larger gaps bet...

...king on results I am not sure how to explain them though. For example r325313 fixes "use after free", it should not give any performance slowdowns or boosts. Though if I read results right, they show 23.65% slowdown for time of linking linux kernel (http://lnt.llvm.org/db_default/v4/link/104). I guess such variation can happen for example if bot do only single link iteration for tests, so that final time is just a error mostly probably. task-clock results are available for "linux-kernel" and "llvm-as-fsds" only and all other tests has blank field. Should it mean...