We've had a lot of churn in all the trunks (llvm, llvm-gcc, clang) recently, and the testing buildbots have been failing repeatedly. I spoke with Chris this AM, and he suggested we have a "stabilization day." Please avoid large, destabilizing changes for about twenty-four hours. We would like for the testing bots to begin working again. Thanks, stuart
On Jul 15, 2009, at 11:52 AMPDT, Stuart Hastings wrote:> We've had a lot of churn in all the trunks (llvm, llvm-gcc, clang) > recently, and the testing buildbots have been failing repeatedly. > > I spoke with Chris this AM, and he suggested we have a "stabilization > day." Please avoid large, destabilizing changes for about twenty-four > hours. We would like for the testing bots to begin working again.OK. I wonder if we might be able to automate the stabilization somewhat. I'm not at all sure this can be done without introducing worse problems that it solves, but here's some discussion fodder: Have the buildbots (or, probably better, one Master Buildbot) do auto- reversion when they see a new failure. They would need to be able to detect bogus failures, such as temporary inability to connect to the svn server. Have checkins go to a branch, and have the buildbots automove them into mainline only after passing regression checks on the branch. If the procedures go wrong I can easily imagine the tree getting into a state where nobody knows what's in it very quickly, so we need to be careful...
On 2009-07-15 23:24, Dale Johannesen wrote:> On Jul 15, 2009, at 11:52 AMPDT, Stuart Hastings wrote: > > >> We've had a lot of churn in all the trunks (llvm, llvm-gcc, clang) >> recently, and the testing buildbots have been failing repeatedly. >> >> I spoke with Chris this AM, and he suggested we have a "stabilization >> day." Please avoid large, destabilizing changes for about twenty-four >> hours. We would like for the testing bots to begin working again. >> > > OK. > I wonder if we might be able to automate the stabilization somewhat. > I'm not at all sure this can be done without introducing worse > problems that it solves, but here's some discussion fodder: > > Have the buildbots (or, probably better, one Master Buildbot) do auto- > reversion when they see a new failure. They would need to be able to > detect bogus failures, such as temporary inability to connect to the > svn server. > > Have checkins go to a branch, and have the buildbots automove them > into mainline only after passing regression checks on the branch. > > If the procedures go wrong I can easily imagine the tree getting into > a state where nobody knows what's in it very quickly, so we need to be > careful... >I'm not too keen about seeing buildbots play with trunk ;) How about starting simple, and just auto-tagging builds that work? Could be done per OS/arch, and one global tag when all buildbots pass. It would also be useful to have a SVN pre/post-commit hook display the number of current buildbot failures, as a reminder that trunk is broken. Best regards, --Edwin
I don't think auto-reversion is ever going to be a very good idea, and don't envy anyone trying to implement it correctly. Usually it is the case that build breakages can be easily fixed, when not it usually the author is most capable of reverting them safely. An alternative approach to improving the current status is to make the buildbots detect failures sooner. Right now the Apple bootstrap buildbot takes two hours. This means a four hour latency between what may be a failure and what may be a fix. A simple fix for this is to add a new buildbot which only tests one particular Apple style build, instead of the 6 or however many it currently builds. - Daniel On Wed, Jul 15, 2009 at 1:24 PM, Dale Johannesen<dalej at apple.com> wrote:> > On Jul 15, 2009, at 11:52 AMPDT, Stuart Hastings wrote: > >> We've had a lot of churn in all the trunks (llvm, llvm-gcc, clang) >> recently, and the testing buildbots have been failing repeatedly. >> >> I spoke with Chris this AM, and he suggested we have a "stabilization >> day." Please avoid large, destabilizing changes for about twenty-four >> hours. We would like for the testing bots to begin working again. > > OK. > I wonder if we might be able to automate the stabilization somewhat. > I'm not at all sure this can be done without introducing worse > problems that it solves, but here's some discussion fodder: > > Have the buildbots (or, probably better, one Master Buildbot) do auto- > reversion when they see a new failure. They would need to be able to > detect bogus failures, such as temporary inability to connect to the > svn server. > > Have checkins go to a branch, and have the buildbots automove them > into mainline only after passing regression checks on the branch. > > If the procedures go wrong I can easily imagine the tree getting into > a state where nobody knows what's in it very quickly, so we need to be > careful... > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >