Dear List, There's been some recent discussion on the list about benchmarks. I just read a Dr. Dobbs article on the relative runtime performance of various compilers (8 of them compared) on Intel platforms. The test focused on mainly template type things but offers Dhrystone and zlib for comparisons. There's no clear winner as all compilers perform well in some areas and poorly in others. The overall rating (Table 2 in the article) ranks the compilers thusly (higher is better): Intel 8.0 9.22 VC++ 7.1 7.56 CodeWarrior 7.44 GCC 3.2 6.67 VC++ 6.0 6.00 Comeau 4.3.3 4.56 Borland 5.6.4 3.78 Open Watcom 1.2 3.00 I am, of course, curious how LLVM stacks up in both native code generation and the CBE. I guess the main point is that GCC is not the compiler to beat, Intel 8.0 is (at least for the template based tests this benchmark was aimed at). Any chance we can get this test suite included in the benchmark? You can find the article online here: http://www.ddj.com/articles/2004/0405/ but you'll need to register. Reid -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20040430/04442a30/attachment.sig>
I think this is a very good idea and we will definitely pursue it. Give us a little time, though. Several of the students are either doing their Ph.D. proposal or preparing to go away for summer internships or both. Adding the benchmarks and reporting initial numbers should not be too difficult, but reporting any meaningful numbers will take some work. --Vikram http://www.cs.uiuc.edu/~vadve http://llvm.cs.uiuc.edu/ On Apr 30, 2004, at 11:36 AM, Reid Spencer wrote:> Dear List, > > There's been some recent discussion on the list about benchmarks. I > just > read a Dr. Dobbs article on the relative runtime performance of various > compilers (8 of them compared) on Intel platforms. The test focused on > mainly template type things but offers Dhrystone and zlib for > comparisons. > > There's no clear winner as all compilers perform well in some areas and > poorly in others. The overall rating (Table 2 in the article) ranks the > compilers thusly (higher is better): > > Intel 8.0 9.22 > VC++ 7.1 7.56 > CodeWarrior 7.44 > GCC 3.2 6.67 > VC++ 6.0 6.00 > Comeau 4.3.3 4.56 > Borland 5.6.4 3.78 > Open Watcom 1.2 3.00 > > I am, of course, curious how LLVM stacks up in both native code > generation and the CBE. I guess the main point is that GCC is not the > compiler to beat, Intel 8.0 is (at least for the template based tests > this benchmark was aimed at). Any chance we can get this test suite > included in the benchmark? > > You can find the article online here: > http://www.ddj.com/articles/2004/0405/ > but you'll need to register. > > Reid
On Fri, 30 Apr 2004, Reid Spencer wrote:> There's been some recent discussion on the list about benchmarks. I just > read a Dr. Dobbs article on the relative runtime performance of various > compilers (8 of them compared) on Intel platforms. The test focused on > mainly template type things but offers Dhrystone and zlib for > comparisons.Interesting, I'll definitely take a look. LLVM should do fairly well on the tests I would guess.> I am, of course, curious how LLVM stacks up in both native code > generation and the CBE. I guess the main point is that GCC is not the > compiler to beat, Intel 8.0 is (at least for the template based tests > this benchmark was aimed at).Yes, definitely. GCC is not the end-goal, and for that matter, neither is ICC. :)> Any chance we can get this test suite included in the benchmark?There is a chance, but as Vikram mentioned, it's not extremely likely to happen in the immediate future. However, if you or someone else wrote the makefiles neccesary to the suite to the llvm/test/Programs hierarchy, we would be happy to add them and have our automated testers run them. :) In particular, I'd really like to try to foster some more community involvement in all aspects of LLVM, including coding, porting, testing, documentation, performance tweaking, ... Our group obviously has a lot invested in LLVM and we will continue doing lots of interesting things, but unfortunately we do have limited people-bandwidth. If you *really* want something done (like any open source project), the best way is to do it yourself of convince someone to do it for you. I'm not try to complain here: we have had a number of high-quality contributions from several different people, I just wanted to point out that asking us to do something isn't the only way to get it done. :-) -Chris -- http://llvm.cs.uiuc.edu/ http://www.nondot.org/~sabre/Projects/
Reid Spencer
2004-May-01 20:04 UTC
[LLVMdev] Open Source Contributions (was Re: Benchmarks)
On Fri, 2004-04-30 at 10:57, Chris Lattner wrote:> There is a chance, but as Vikram mentioned, it's not extremely likely to > happen in the immediate future. However, if you or someone else wrote the > makefiles neccesary to the suite to the llvm/test/Programs hierarchy, we > would be happy to add them and have our automated testers run them. :) > > In particular, I'd really like to try to foster some more community > involvement in all aspects of LLVM, including coding, porting, testing, > documentation, performance tweaking, ... Our group obviously has a lot > invested in LLVM and we will continue doing lots of interesting things, > but unfortunately we do have limited people-bandwidth. If you *really* > want something done (like any open source project), the best way is to do > it yourself of convince someone to do it for you. > > I'm not try to complain here: we have had a number of high-quality > contributions from several different people, I just wanted to point out > that asking us to do something isn't the only way to get it done. :-) > > -ChrisI'd be happy to contribute more and in fact, there are lots of things I could and would contribute. Pretty much the only thing that stops me is the project's CVS policy. Submitting patches is fine for smaller tasks (single file bug fixes etc.) but larger tasks (like adding a whole test suite) really need CVS write access to be done efficiently and correctly. Perhaps its just me, but I don't have time to dicker with patch creation, wait for individual files to be added, processing emails that let me know when things have been added, wait for further responses on modifications, etc. I have a full-time job, a full-time family, and a full-time second career (erm, perhaps I'm just over-booked? :) I need some efficiency if I'm to do anything. From my perspective, the process is _way_ more efficient if I can just create a branch, do what I need to do and then tell you "look at branch xyz and merge if you like it". Its possible with CVS to restrict access to main line commits to certain users so you can still maintain control of the main development trend. Furthermore, providing CVS access should reduce your administrative burden. Instead of patching a pile of individual files, you can simply look at the changes introduced by a branch and decide if its something you want to keep or inform the branch author of the things that need to be modified. You can also set the "gold standard" for contributions to make sure that (a) branches are maintained with the mainline by the author (thereby reducing mainline conflicts on merge), (b) the software on the branch must build correctly, (c) the branch software must be well integrated into the build system, (d) the branch software must not break any existing tests, (e) etc., etc. I understand the University has certain legal restrictions about granting access to non-student and non-faculty users. That may be the trump card that prevents wider use of CVS by contributors to LLVM. If that is the case, I would suggest that (a) the project simply accept that contributions from others will be minimal or (b) move the CVS repository somewhere that doesn't have the University's restrictions. That last option, however, may have additional intellectual property issues. While it would be unwise to freely grant write access to the CVS repository to anyone that asked for it, you might want to think about some qualifications necessary to allow that to happen in a controlled fashion. I for one don't have any problems being asked to qualify for CVS write access. If such access were available to serious and capable contributors, I believe you'd get a lot more contributions (as I've seen on other projects). Furthermore, the contributions are generally of a higher quality because the technical requirements go up. Its something of a self-sorting process. Ultimately the decision is yours. LLVM is still great either way. :) Reid. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20040501/a5ff2573/attachment.sig>
On Fri, 30 Apr 2004, Reid Spencer wrote:> Dear List, > > There's been some recent discussion on the list about benchmarks. I just > read a Dr. Dobbs article on the relative runtime performance of various > compilers (8 of them compared) on Intel platforms. The test focused on > mainly template type things but offers Dhrystone and zlib for > comparisons.I bought the issue and took a look. I suspect that LLVM will do extremely well on these tests, but it doesn't look like there is a publically available download for his benchmarks. I'm going to email the author and see if we can get a copy. -Chris -- http://llvm.cs.uiuc.edu/ http://www.nondot.org/~sabre/Projects/
On Sat, 2004-05-01 at 22:30, Chris Lattner wrote:> On Fri, 30 Apr 2004, Reid Spencer wrote: > > > Dear List, > > > > There's been some recent discussion on the list about benchmarks. I just > > read a Dr. Dobbs article on the relative runtime performance of various > > compilers (8 of them compared) on Intel platforms. The test focused on > > mainly template type things but offers Dhrystone and zlib for > > comparisons. > > I bought the issue and took a look. I suspect that LLVM will do extremely > well on these tests, but it doesn't look like there is a publically > available download for his benchmarks. I'm going to email the author and > see if we can get a copy. > > -ChrisI was looking for that too. The author has a web site for errata and updates but there's nothing on it. Hopefully DDJ won't be too worried about the IP. Reid. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20040502/950bdcfd/attachment.sig>