On 2009-07-15 23:24, Dale Johannesen wrote:> On Jul 15, 2009, at 11:52 AMPDT, Stuart Hastings wrote: > > >> We've had a lot of churn in all the trunks (llvm, llvm-gcc, clang) >> recently, and the testing buildbots have been failing repeatedly. >> >> I spoke with Chris this AM, and he suggested we have a "stabilization >> day." Please avoid large, destabilizing changes for about twenty-four >> hours. We would like for the testing bots to begin working again. >> > > OK. > I wonder if we might be able to automate the stabilization somewhat. > I'm not at all sure this can be done without introducing worse > problems that it solves, but here's some discussion fodder: > > Have the buildbots (or, probably better, one Master Buildbot) do auto- > reversion when they see a new failure. They would need to be able to > detect bogus failures, such as temporary inability to connect to the > svn server. > > Have checkins go to a branch, and have the buildbots automove them > into mainline only after passing regression checks on the branch. > > If the procedures go wrong I can easily imagine the tree getting into > a state where nobody knows what's in it very quickly, so we need to be > careful... >I'm not too keen about seeing buildbots play with trunk ;) How about starting simple, and just auto-tagging builds that work? Could be done per OS/arch, and one global tag when all buildbots pass. It would also be useful to have a SVN pre/post-commit hook display the number of current buildbot failures, as a reminder that trunk is broken. Best regards, --Edwin
On Jul 15, 2009, at 1:43 PMPDT, Török Edwin wrote:> On 2009-07-15 23:24, Dale Johannesen wrote: >> On Jul 15, 2009, at 11:52 AMPDT, Stuart Hastings wrote:>> I wonder if we might be able to automate the stabilization somewhat. >> I'm not at all sure this can be done without introducing worse >> problems that it solves, but here's some discussion fodder: >> >> Have the buildbots (or, probably better, one Master Buildbot) do >> auto- >> reversion when they see a new failure. They would need to be able to >> detect bogus failures, such as temporary inability to connect to the >> svn server. >> >> Have checkins go to a branch, and have the buildbots automove them >> into mainline only after passing regression checks on the branch. >> >> If the procedures go wrong I can easily imagine the tree getting into >> a state where nobody knows what's in it very quickly, so we need to >> be >> careful... >> > > I'm not too keen about seeing buildbots play with trunk ;)Nor am I; normally I'd be the last person to suggest something like this. But in the last few days we've seen just how bad a job humans can do...> How about starting simple, and just auto-tagging builds that work? > Could be done per OS/arch, and one global tag when all buildbots pass. > > It would also be useful to have a SVN pre/post-commit hook display the > number of current buildbot failures, > as a reminder that trunk is broken. > > Best regards, > --Edwin > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
On Wed, Jul 15, 2009 at 1:50 PM, Dale Johannesen<dalej at apple.com> wrote:> > On Jul 15, 2009, at 1:43 PMPDT, Török Edwin wrote: >> On 2009-07-15 23:24, Dale Johannesen wrote: >>> On Jul 15, 2009, at 11:52 AMPDT, Stuart Hastings wrote: > >>> I wonder if we might be able to automate the stabilization somewhat. >>> I'm not at all sure this can be done without introducing worse >>> problems that it solves, but here's some discussion fodder: >>> >>> Have the buildbots (or, probably better, one Master Buildbot) do >>> auto- >>> reversion when they see a new failure. They would need to be able to >>> detect bogus failures, such as temporary inability to connect to the >>> svn server. >>> >>> Have checkins go to a branch, and have the buildbots automove them >>> into mainline only after passing regression checks on the branch. >>> >>> If the procedures go wrong I can easily imagine the tree getting into >>> a state where nobody knows what's in it very quickly, so we need to >>> be >>> careful... >>> >> >> I'm not too keen about seeing buildbots play with trunk ;) > > Nor am I; normally I'd be the last person to suggest something like > this. But in the last few days we've seen just how bad a job humans > can do... >We would need a much much more sophisticated testing system before we can do something automated with reverting patches. One unfortunate side-effect of auto-reverting is that it could revert one patch that has a fix for that patch in the pipeline. Leading to unnecessary churn by the build bots. The core problem, in my opinion, is that people *don't* pay attention to the build bot failure messages that come along. This is as much a systemic problem as it is a community problem. Community in that we need to foster the submission of well-tested patches. Systemic because this is not how people hacking on open source projects typically develop projects. I like Daniel's idea of throwing more machines at the problem. It's a brute force method, but there you go. -bw
On Wednesday 15 July 2009 15:43, Török Edwin wrote:> How about starting simple, and just auto-tagging builds that work? > Could be done per OS/arch, and one global tag when all buildbots pass.We've talked about this before and I've been working on setting up such a system. Unfortunately, I can't figure out why my buildbots fail to configure llvm-gcc. Is there a link to the buildbots on the website? I can't find it. -Dave
On 2009-07-16 17:42, David Greene wrote:> On Wednesday 15 July 2009 15:43, Török Edwin wrote: > > >> How about starting simple, and just auto-tagging builds that work? >> Could be done per OS/arch, and one global tag when all buildbots pass. >> > > We've talked about this before and I've been working on setting up > such a system. Unfortunately, I can't figure out why my buildbots > fail to configure llvm-gcc. >I didn't know that it still fails. Can you send me your full buildbot config? I'll try to have a look if I can reproduce the failure on my machine.> Is there a link to the buildbots on the website? I can't find it. >The google buildbots are here, I don't see any link on llvm.org to it: http://google1.osuosl.org:8011/waterfall Best regards, --Edwin
On Thu, Jul 16, 2009 at 7:42 AM, David Greene<dag at cray.com> wrote:> On Wednesday 15 July 2009 15:43, Török Edwin wrote: > >> How about starting simple, and just auto-tagging builds that work? >> Could be done per OS/arch, and one global tag when all buildbots pass. > > We've talked about this before and I've been working on setting up > such a system. Unfortunately, I can't figure out why my buildbots > fail to configure llvm-gcc.I plan to check my buildbot configuration "stuff" into the llvm-project repository soon. I have to refactor things a bit to support use by other people, but currently I have configurations for llvm, clang, and llvm-gcc in various target / self-hosting permutations. I also have a regular buildbot driver nightlytest (the smoosh-01 results on llvm-testresults). My hope is that once checked in Danny and I can merged the current public buildbot configuration and my configuration so that all buildbots are driven from the same master configuration (with additional local configuration magic). This should make it easier for other organizations to set up their own buildbots, and to add new buildbot configurations (like MSVC/cmake).> Is there a link to the buildbots on the website? I can't find it.Chris updated the website, its now listed under "Dev. Resources". Please consider contributing a buildslave for the platform you care about. - Daniel> > -Dave > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >
2009/7/15 Török Edwin <edwintorok at gmail.com>:> I'm not too keen about seeing buildbots play with trunk ;) > > How about starting simple, and just auto-tagging builds that work? > Could be done per OS/arch, and one global tag when all buildbots pass.I don't know anything about svn performance. Would this negatively impact llvm.org, which is already pretty strained? If this is reasonable I think I can update my buildbots to do this fairly easily. The global tag is difficult since it requires coordination, but individual tags would be easy. It would be nice to get the mails from them filtered out, otherwise it will make the commit list very noisy.> It would also be useful to have a SVN pre/post-commit hook display the > number of current buildbot failures, > as a reminder that trunk is broken.I think this would have a negative performance impact, at least until llvm.org is running the public buildmaster. - Daniel> Best regards, > --Edwin > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >
On Jul 16, 2009, at 12:04 PM, Daniel Dunbar wrote:> 2009/7/15 Török Edwin <edwintorok at gmail.com>: >> I'm not too keen about seeing buildbots play with trunk ;) >> >> How about starting simple, and just auto-tagging builds that work? >> Could be done per OS/arch, and one global tag when all buildbots >> pass. > > I don't know anything about svn performance. Would this negatively > impact llvm.org, which is already pretty strained? >How is this any different then the buildbots displaying the revision tag for a build that works? And the definition of "works" I assume you mean build and make check cleanly. llvm-test is a different beast and you need to compare from one run to the next because we do not currently pass for everything (which is the job of the nightly tester in theory). Also, llvm.org is severely strained right now. We are upgrading the server but I need a few weeks for this. If its possible, it would be nice to have the buildbots update and not do a full check out. -Tanya> If this is reasonable I think I can update my buildbots to do this > fairly easily. The global tag is difficult since it requires > coordination, but individual tags would be easy. It would be nice to > get the mails from them filtered out, otherwise it will make the > commit list very noisy. > >> It would also be useful to have a SVN pre/post-commit hook display >> the >> number of current buildbot failures, >> as a reminder that trunk is broken. > > I think this would have a negative performance impact, at least until > llvm.org is running the public buildmaster. > > - Daniel > >> Best regards, >> --Edwin >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >> > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20090716/79e35060/attachment.html>
On Thursday 16 July 2009 14:04, Daniel Dunbar wrote:> 2009/7/15 Török Edwin <edwintorok at gmail.com>: > > I'm not too keen about seeing buildbots play with trunk ;) > > > > How about starting simple, and just auto-tagging builds that work? > > Could be done per OS/arch, and one global tag when all buildbots pass. > > I don't know anything about svn performance. Would this negatively > impact llvm.org, which is already pretty strained?Tags are cheap.> If this is reasonable I think I can update my buildbots to do this > fairly easily. The global tag is difficult since it requires > coordination, but individual tags would be easy. It would be nice to > get the mails from them filtered out, otherwise it will make the > commit list very noisy.Thanks to Törok, I think I have the validator buildbot working. The recipe I have does this: * Build llvm * Run make check on llvm * Build llvm-gcc, pointed to the just-built llvm * Install llvm-gcc in a temporary locationo * Build llvm again, pointing to the just-installed llvm-gcc * Run make check on llvm * Run llvm-test If all of these pass, the validation is considered to succeed. I can easily add make check on llvm-gcc if/when we support it. I'd like to take advantage of any tagging infrastructure you add. It would be nice to auto-tag individual validation builds as working. I agree that global tagging can't be automated within the buildbot. Perhaps a daemon could listed to e-mails from the buildbots and tag when a particular revision passes all of the buildbot. That'll come later, though. In the meantime, should I attempt to merge my recipe into your buildbot config when you check it into the llvm repository? I'd make separate buildbot factories to run the two recipes (nightly and validation) but the smae slave pool could be used for both. Or should these remain as completely separate buildbots? One long-term goal for me is to make validation even more target-sensitive. So for example if we had a way to set default llvm and/or llvm-gcc options to compile for specific implementations (Barcelona vs. Nehalem, for example), we could more rigorously test the optimizer and code generator. I know that for us this would be a big win. But AFAIK there's no way to set default compiler flags like this right now. -Dave