> Be careful comparing these numbers. I see that we have a 23x speedup > today over GCC on the "burg" test, but we go from 0.093 -> 0.004s. :) > The shorter the test runs get, the more noisy they get, so unfortunately > we're not getting a realistic 23x speedup here. ;-) > [...] > These are even more dubious. In particular, only the first 6 rows contain > programs with reasonable runtimes. This means that the 7x speedups for > going from 0.021 -> 0.003 don't really count. :) > [...] > This one is just noise, if you look today it's 1.0's straight across the > board. Also note that the test runs for 0.003 seconds, which is the > resolution of the time command on the system the program is being run on: > this is not a good test for checking performance. :)yes, i saw, that the numbers are quite discrete, it is just another feature telling the same. Well, but who really needs insecure numbers?.. maybe it would be a good idea to change the test a bit or make the same test but in a loop? otherwize it makes no sense to include them into benchmarking...> Hrm, you're not happy with the 2.25x speedup we get now?opss, sorry, my mistake.> With GCC it > takes 9.639s to run the test, with LLVM-CBE it takes 4.3s, and with > LLVM-LLC-LS it takes 4.963s. I think these are pretty good numbers. :)oh, you are right, i like those numbers :)> Actually we spend *VERY* little time tuning and tweaking the optimizer for > performance. Something that would be *INCREDIBLY* useful would be for > someone to pick some benchmark or other program we do poorly on (e.g. > Ptrdist-ks), and find out *WHAT* we could be doing to improve it. A good > way to do this is to take the program, run it in a profiler (llvm-prof or > gprof) find the hot spots, and see what we're code generating for them, > and suggest ways that it could be improved. If something performs well > with the CBE but not with LLC-LS, then compare the native machine code > generated, if it performs poorly with both, then it's probably an LLVM > optimization. > > At this point there are a huge number of possibilities for improvement. > We have very little in the way of loop optimizations, and we don't > actually use an interprocedural pointer analysis (I'm hope to rectify this > for 1.3, it should make a huge difference). Even if you're not into > hacking on LLVM optimizations, identifying code that we could improve > (and reducing them down to small examples of code we compile poorly) is > incredibly useful. > > Consider this a small plea for help. :) Once we know what to fix, it's > usually pretty easy to do so, but identifying the problems takes time, and > we have plenty of other things we need to be doing as well.well, 90+ % of my spare time is dedicated to SuSE 9 and win32/cygwin. Neither of two is good for LLVM today. But me and others always have a web client! Would it be a big deal to add small functionality to the http://llvm.cs.uiuc.edu/demo/ page? I mean, why not to add output in assembler there? Then everyone could try to help you from anywhere. If i remember right, Misha was the one who prepared this nice page, could we ask Misha for that great add-on? with best regards, Valery.
On Fri, 30 Apr 2004, [koi8-r] "Valery A.Khamenya[koi8-r] " wrote:> > This one is just noise, if you look today it's 1.0's straight across the > > board. Also note that the test runs for 0.003 seconds, which is the > > resolution of the time command on the system the program is being run on: > > this is not a good test for checking performance. :) > > yes, i saw, that the numbers are quite discrete, it is just another > feature telling the same. Well, but who really needs insecure numbers?.. > maybe it would be a good idea to change the test a bit or make the > same test but in a loop? otherwize it makes no sense to include them > into benchmarking...The nightly tester is used for two purposes: making sure that nothing breaks (the unit tests) and keeping tabs on how well performance is doing (the spec and most multisource tests). It's not a reliable way to do serious benchmarking, but can give good insights into where things can be improved.> > With GCC it > > takes 9.639s to run the test, with LLVM-CBE it takes 4.3s, and with > > LLVM-LLC-LS it takes 4.963s. I think these are pretty good numbers. :) > > oh, you are right, i like those numbers :):)> > Consider this a small plea for help. :) Once we know what to fix, it's > > usually pretty easy to do so, but identifying the problems takes time, and > > we have plenty of other things we need to be doing as well. > > well, 90+ % of my spare time is dedicated to SuSE 9 and win32/cygwin. > Neither of two is good for LLVM today.What is wrong with SuSE 9? Is it the wierd GCC ICE bug? If so, I believe that it only effects one .cpp file in the LLVM sources and can probably be worked around. I don't have a copy of the buggy compiler handy, but if you were to look into it it would probably be a pretty easy problem to solve.> But me and others always have a web client! Would it be a big deal to > add small functionality to the http://llvm.cs.uiuc.edu/demo/ page? I > mean, why not to add output in assembler there? Then everyone could try > to help you from anywhere. If i remember right, Misha was the one who > prepared this nice page, could we ask Misha for that great add-on?It would certainly be possible, but the point of the demo page is to show how the C front-end and optimizer transform programs to LLVM code. I might be able to con Brian or Misha into adding support for native code generation, but we'll have to see (they're all really busy with end-of-semester stuff). -Chris -- http://llvm.cs.uiuc.edu/ http://www.nondot.org/~sabre/Projects/
> The nightly tester is used for two purposes: making sure that nothing > breaks (the unit tests) and keeping tabs on how well performance is doing > (the spec and most multisource tests). It's not a reliable way to do > serious benchmarking, but can give good insights into where things can be > improved.hm, one day, the great benchmarking will be a reason to use LLVM, so, i think a bit more benchmarking info could be fruitful. If test might be enveloped in a loop to get run about few seconds then it makes hardly nightly testings much slower, but will bring a lot of interesting info.> What is wrong with SuSE 9? Is it the wierd GCC ICE bug? If so, I believe > that it only effects one .cpp file in the LLVM sources and can probably be > worked around. I don't have a copy of the buggy compiler handy, but if > you were to look into it it would probably be a pretty easy problem to > solve.yesterday I got new SuSE 9.1 DVD, so i am going to enter this river again. Perhaps, this time all will be fine.> It would certainly be possible, but the point of the demo page is to show > how the C front-end and optimizer transform programs to LLVM code. I > might be able to con Brian or Misha into adding support for native code > generation, but we'll have to see (they're all really busy with > end-of-semester stuff).Such an add-on would be really nice. BTW, my good wishes to them concerning end-of-semester. -- Valery