search for: ministat

On 06/30/2013 02:14 AM, Anton Korobeynikov wrote: > Hi Tobi, > > First of all, all this is http://llvm.org/bugs/show_bug.cgi?id=1367 :) > >> The statistical test ministat is performing seems simple and pretty >> standard. Is there any reason we could not do something similar? Or are we >> doing it already and it just does not work as expected? > The main problem with such sort of tests is that we cannot trust them, unless: > 1. The data has the no...

...data of the original run, we get: ~$ cat /tmp/data-before 0.0120 0.0080 0.0200 ~$ cat /tmp/data-after 0.0200 0.0240 0.0200 It seems there is a lot of noise involved. Still, LNT is reporting this result without understanding that the results for this benchmark are unreliable. In contrast, the ministat [2] tool is perfectly capable of understanding that those results are insufficient to prove any statistical difference at 90% confidence. ======================================================================= $ ./src/ministat -c 90 /tmp/data-before /tmp/data-after x /tmp/data-before + /tmp/data-a...

Hi Tobias, > I trust your knowledge about statistics, but am wondering why ministat (and > it's t-test) is promoted as a statistical sane tool for benchmarking > results. I do not know... Ask author of ministat? > Is the use of the t-test for benchmark results a bad idea in > general? No, in general. But one should be aware about the assumptions of the underlying...

Hi Tobi, First of all, all this is http://llvm.org/bugs/show_bug.cgi?id=1367 :) > The statistical test ministat is performing seems simple and pretty > standard. Is there any reason we could not do something similar? Or are we > doing it already and it just does not work as expected? The main problem with such sort of tests is that we cannot trust them, unless: 1. The data has the normal distribution 2...

On 28 June 2013 19:45, Chris Matthews <chris.matthews at apple.com> wrote: > Given this tradeoff I think we want to tend towards false positives (over > false negatives) strictly as a matter of compiler quality. > False hits are not binary, but (at least) two-dimensional. You can't say it's better to have any amount of false positives than any amount of false negatives

I should describe the cost of false negatives and false positives, since I think it matters for how this problem is approached. False negatives mean we miss a real regression --- we don’t want that. False positives mean somebody has to spend some time looking at and reproducing the regression when there is not one --- bad too. Given this tradeoff I think we want to tend towards false positives

...my system I get 0.06 sec for Polly-load 0.09 sec for Polly-optimize What exact version of Polybench did you use? What compiler flags did you use to compile the benchmark? Also, did you run the executables several times? How large is the standard deviation of the results? (You can use a tool like ministat to calculate these values [1]) > | gramchmidt.c | 0.159 | 0.167 | 1.023 | 5.0% | 543.4% | > | eidel.c | 0.125 | 0.13 | 0.285 | 4.0% | 128.0% | > | adi.c | 0.155 | 0.156 | 0.953 | 0.6% | 514.8% | > | doitgen.c | 0.124 | 0.128 | 0.298 | 3.2% | 140.3% | > | intrument.c | 0.149 | 0.151...

...my system I get 0.06 sec for Polly-load 0.09 sec for Polly-optimize What exact version of Polybench did you use? What compiler flags did you use to compile the benchmark? Also, did you run the executables several times? How large is the standard deviation of the results? (You can use a tool like ministat to calculate these values [1]) > | gramchmidt.c | 0.159 | 0.167 | 1.023 | 5.0% | 543.4% | > | eidel.c | 0.125 | 0.13 | 0.285 | 4.0% | 128.0% | > | adi.c | 0.155 | 0.156 | 0.953 | 0.6% | 514.8% | > | doitgen.c | 0.124 | 0.128 | 0.298 | 3.2% | 140.3% | > | intrument.c | 0.149 | 0.151...

...rameters for expressions {p + 1, +, 1}_<l> and {p + 2, +, 1}_<l> in case the loop 'l' is not part of the scop. > Our evaluation is based on Intel Pentium Dual CPU T2390(1.86GHz) with > 2GB DDR2 memory. Each benchmark is run multiple times and data are > collected using ministat (https://github.com/codahale/ministat). > Polly is built with Cloog in Release+Assert mode. All Polly, LLVM and > Cloog source code are checked out from SVN at April 24, 2013. Can you give the svn revision / git hash for the checkouts of LLVM, Polly, isl, cloog and polybench. > Table 1 a...

Dear Tobias, Thank you very much for your very helpful advice. Yes, -debug-pass and -time-passes are two very useful and powerful options when evaluating the compile-time of each compiler pass. They are exactly what I need! With these options, I can step into details of the compile-time overhead of each pass. I have finished some preliminary testing based on two randomly selected files from

Why does GCC produce faster code using "-march=pentium2 -mtune=pentium4" on a Pentium 4 chip versus plain -march=pentium4? Try it... CPUTYPE=pentium2 CFLAGS+= -mtune=pentium4 COPTFLAGS+= -mtune=pentium4 -- BSD Podcasts @ http://bsdtalk.blogspot.com/

On 04/30/2013 04:13 PM, Star Tan wrote: > Hi all, [...] > How could I find out where the time is spent on between two adjacent Polly passes? Can anyone give me some advice? Hi Star Tan, I propose to do the performance analysis using the 'opt' tool and optimizing LLVM-IR, instead of running it from within clang. For the 'opt' tool there are two commands that should help

...large compile time increase can be seen for mediabench. Reaching this goal would be a great step forward. > [Attachments] > > Our evaluation is based on Intel Pentium Dual CPU T2390(1.86GHz) with > 2GB DDR2 memory. Each benchmark is run multiple times and data are > collected using ministat (https://github.com/codahale/ministat). Results > are shown in table 1 and table 2. Five cases are tested: (alias > pollycc="clang -O3 -load LLVMPolly.so -mllvm -polly) *clang: clang -O3 > *pLoad: clang -O3 -load LLVMPolly.so *pNoGen:pollycc -O3 -mllvm > -polly-optimizer=none -mll...

Dear Tobias Grosser, Today I have rebuilt the LLVM-Polly in Release mode. The configuration of my own testing machine is: Intel Pentium Dual CPU T2390(1.86GHz) with 2GB DDR2 memory. I evaluated the Polly using PolyBench and Mediabench. It takes too long time to evaluate the whole LLVM-testsuite, so I just choose the Mediabench from LLVM-testsuite. The preliminary results of Polly compiling

...>{p + 1, +, 1}_<l> and {p + 2, +, 1}_<l> in case the loop 'l' is not part >of the scop. > >> Our evaluation is based on Intel Pentium Dual CPU T2390(1.86GHz) with >> 2GB DDR2 memory. Each benchmark is run multiple times and data are >> collected using ministat (https://github.com/codahale/ministat). >> Polly is built with Cloog in Release+Assert mode. All Polly, LLVM and >> Cloog source code are checked out from SVN at April 24, 2013. > >Can you give the svn revision / git hash for the checkouts of LLVM, >Polly, isl, cloog and polybe...

Hi all, I have updated my GSoS proposal: "FastPolly: Reducing LLVM-Polly Compiling overhead" (https://gist.github.com/tanstar/5441808). I think the pass ordering problem you discussed early can be also investigated in this project! Is there any comment or advice about my proposal? I appreciate all your help and advice. Thanks, Star Tan Proposal:

Dear Tobias Grosser, Thank you so much for your kind reply. Your advice is very helpful and inspiring. At 2013-03-18 20:40:50,"Tobias Grosser" <tobias at grosser.es> wrote: >On 03/17/2013 11:54 PM, Star Tan wrote: >> Hello Tobi, >> >> I am interested in Polly project. Polly seems to be a very promising tool to find out program parallelization based on LLVM

...clusion list is provided below. I constructed this exclusion list primarily based on the following experiment: I ran the test suite 10 times in three configurations: 1) On an IBM POWER7 (P7) with -O3 -mvsx, 2) On a P7 at -O0 and 3) On an Intel Xeon E5430 with -O3 all using make -j6. I then used the ministat utility (which performs a T test) to compare the timings of the two P7 configurations against each other and the Xeon configuration, requiring a detectable difference at 99.5% confidence. I looked for tests that showed no significant difference in all three comparisons. The running configuration he...

...69) 0.0349051 case 2x: 778.622 (1092.68) 129.342 (156.324) 83.3883 case 3x: 2399.11 (3548.32) 114.913 (139.5) 95.2102 2) pgbench: pgbench version: http://www.postgresql.org/ftp/snapshot/dev/ tool used for benchmarking: git://git.postgresql.org/git/pgbench-tools.git Ananlysis is done using ministat. Test is done for 1x overcommit to check overhead of pv spinlock. There is small performance penalty in non contention scenario (note BASE is jeremy's ticketlock). But with increase in number of threads, improvement is seen. guest: 64bit 8 vCPU and 8GB RAM shared buffer size = 2GB x base_kern...

...69) 0.0349051 case 2x: 778.622 (1092.68) 129.342 (156.324) 83.3883 case 3x: 2399.11 (3548.32) 114.913 (139.5) 95.2102 2) pgbench: pgbench version: http://www.postgresql.org/ftp/snapshot/dev/ tool used for benchmarking: git://git.postgresql.org/git/pgbench-tools.git Ananlysis is done using ministat. Test is done for 1x overcommit to check overhead of pv spinlock. There is small performance penalty in non contention scenario (note BASE is jeremy's ticketlock). But with increase in number of threads, improvement is seen. guest: 64bit 8 vCPU and 8GB RAM shared buffer size = 2GB x base_kern...