On Sat, 5 May 2007, David Greene wrote:> There's the llvm-testresults list, but I find it less than fully useful > because it's not immediately obvious from scanning message subjects if > there's been a test failure. It's a lot of messages to wade through and > read to get this information.Right.> What about a Tinderbox-like setup where we could consult a web page to > see the current status of the repository? Boost has a nice setup: > > http://engineering.meta-comm.com/boost-regression/CVS-RC_1_34_0/developer/index_release.html > > It's probably more complex than what we need. Maybe we just need a > page grouping each test under it's suite and marking the result on > each architecture. Something like this:I think that this is a great idea. However, instead of picking up yet another setting of testing infrastructure, I think we should make what we have already (the nightly testers) better. In particular, the main page of the tester: http://llvm.org/nightlytest/ Already captures a lot of this: it tells you the number of unexpected failures, whether or not the build succeeded etc. You can even drill down to a specific machine, e.g.: http://llvm.org/nightlytest/machine.php?machine=120 I see several problems with this, all of which are solvable: 1. The tester script doesn't update and rebuild the CFE, so often you get failures due to an out-of-date CFE. 2. There is no way to group by architecture, target-triple, etc. 3. Minor stupid stuff: on the per-machine page, you can click on the test failures number and go to the list of unexpected failures, but you can't do that on the main page. -Chris -- http://nondot.org/sabre/ http://llvm.org/
Hi Chris,> I think that this is a great idea. However, instead of picking up yet > another setting of testing infrastructure, I think we should make what > we have already (the nightly testers) better. In particular, the main > page of the tester: > http://llvm.org/nightlytest/ > > Already captures a lot of this: it tells you the number of unexpected > failures, whether or not the build succeeded etc. You can even drill > down to a specific machine, e.g.: > http://llvm.org/nightlytest/machine.php?machine=120 > > I see several problems with this, all of which are solvable: > > 1. The tester script doesn't update and rebuild the CFE, so often you > get failures due to an out-of-date CFE. > 2. There is no way to group by architecture, target-triple, etc. > 3. Minor stupid stuff: on the per-machine page, you can click on the > test failures number and go to the list of unexpected failures, but > you can't do that on the main page.Are you aware of buildbot? It's quite widely used and flexible. http://buildbot.sourceforge.net/ "By automatically rebuilding and testing the tree each time something has changed, build problems are pinpointed quickly, before other developers are inconvenienced by the failure. The guilty developer can be **identified and harassed without human intervention**. By running the builds on a variety of platforms, developers who do not have the facilities to test their changes everywhere before checkin will at least know shortly afterwards whether they have broken the build or not. Warning counts, lint checks, image size, compile time, and other build parameters can be tracked over time, are more visible, and are therefore easier to improve." ** -- my emphasis. I'd suggest at least one machine given over to always building whenever anything changes, and have the other nightly volunteers as now. Using just nightly volunteers isn't great because breakage is noticed too late and can affect too many people by then. An example of the output is http://www.python.org/dev/buildbot/trunk/ Cheers, Ralph.
On Sun, 6 May 2007, Ralph Corderoy wrote:> Are you aware of buildbot? It's quite widely used and flexible. > http://buildbot.sourceforge.net/ > I'd suggest at least one machine given over to always building whenever > anything changes, and have the other nightly volunteers as now. Using > just nightly volunteers isn't great because breakage is noticed too late > and can affect too many people by then.That looks nifty. If someone wanted to set it up on their machine, we could give it a try and see how it works. -Chris -- http://nondot.org/sabre/ http://llvm.org/