On Wed, Jul 15, 2009 at 1:50 PM, Dale Johannesen<dalej at apple.com> wrote:> > On Jul 15, 2009, at 1:43 PMPDT, Török Edwin wrote: >> On 2009-07-15 23:24, Dale Johannesen wrote: >>> On Jul 15, 2009, at 11:52 AMPDT, Stuart Hastings wrote: > >>> I wonder if we might be able to automate the stabilization somewhat. >>> I'm not at all sure this can be done without introducing worse >>> problems that it solves, but here's some discussion fodder: >>> >>> Have the buildbots (or, probably better, one Master Buildbot) do >>> auto- >>> reversion when they see a new failure. They would need to be able to >>> detect bogus failures, such as temporary inability to connect to the >>> svn server. >>> >>> Have checkins go to a branch, and have the buildbots automove them >>> into mainline only after passing regression checks on the branch. >>> >>> If the procedures go wrong I can easily imagine the tree getting into >>> a state where nobody knows what's in it very quickly, so we need to >>> be >>> careful... >>> >> >> I'm not too keen about seeing buildbots play with trunk ;) > > Nor am I; normally I'd be the last person to suggest something like > this. But in the last few days we've seen just how bad a job humans > can do... >We would need a much much more sophisticated testing system before we can do something automated with reverting patches. One unfortunate side-effect of auto-reverting is that it could revert one patch that has a fix for that patch in the pipeline. Leading to unnecessary churn by the build bots. The core problem, in my opinion, is that people *don't* pay attention to the build bot failure messages that come along. This is as much a systemic problem as it is a community problem. Community in that we need to foster the submission of well-tested patches. Systemic because this is not how people hacking on open source projects typically develop projects. I like Daniel's idea of throwing more machines at the problem. It's a brute force method, but there you go. -bw
On Jul 15, 2009, at 3:01 PM, Bill Wendling wrote:> On Wed, Jul 15, 2009 at 1:50 PM, Dale Johannesen<dalej at apple.com> > wrote: >> >> On Jul 15, 2009, at 1:43 PMPDT, Török Edwin wrote: >>> On 2009-07-15 23:24, Dale Johannesen wrote: >>>> On Jul 15, 2009, at 11:52 AMPDT, Stuart Hastings wrote: >> >>>> I wonder if we might be able to automate the stabilization >>>> somewhat. >>>> I'm not at all sure this can be done without introducing worse >>>> problems that it solves, but here's some discussion fodder: >>>> >>>> Have the buildbots (or, probably better, one Master Buildbot) do >>>> auto- >>>> reversion when they see a new failure. They would need to be >>>> able to >>>> detect bogus failures, such as temporary inability to connect to >>>> the >>>> svn server. >>>> >>>> Have checkins go to a branch, and have the buildbots automove them >>>> into mainline only after passing regression checks on the branch. >>>> >>>> If the procedures go wrong I can easily imagine the tree getting >>>> into >>>> a state where nobody knows what's in it very quickly, so we need to >>>> be >>>> careful...This seems like the right juncture to remind everyone of Wyland's First Law of Automation: Anything that can be done /for/ you, automatically, can be done /to/ you, automatically. ;-)>>> I'm not too keen about seeing buildbots play with trunk ;) >> >> Nor am I; normally I'd be the last person to suggest something like >> this. But in the last few days we've seen just how bad a job humans >> can do... >> > We would need a much much more sophisticated testing system before we > can do something automated with reverting patches. One unfortunate > side-effect of auto-reverting is that it could revert one patch that > has a fix for that patch in the pipeline. Leading to unnecessary churn > by the build bots. > > The core problem, in my opinion, is that people *don't* pay attention > to the build bot failure messages that come along. This is as much a > systemic problem as it is a community problem. Community in that we > need to foster the submission of well-tested patches. Systemic because > this is not how people hacking on open source projects typically > develop projects. > > I like Daniel's idea of throwing more machines at the problem. It's a > brute force method, but there you go.Adding more buildbots sounds good to me. stuart
On Wed, Jul 15, 2009 at 3:01 PM, Bill Wendling<isanbard at gmail.com> wrote:> The core problem, in my opinion, is that people *don't* pay attention > to the build bot failure messages that come along.That's largely because of the number of false positives. -Eli
On Wed, Jul 15, 2009 at 3:43 PM, Eli Friedman<eli.friedman at gmail.com> wrote:> On Wed, Jul 15, 2009 at 3:01 PM, Bill Wendling<isanbard at gmail.com> wrote: >> The core problem, in my opinion, is that people *don't* pay attention >> to the build bot failure messages that come along. > > That's largely because of the number of false positives. >There have been fewer and fewer of these in recent times. -bw