On Jan 28, 2009, at 8:25 PM, Tanya Lattner wrote:> On Jan 28, 2009, at 3:46 PM, David Greene wrote: > >> On Wednesday 28 January 2009 15:59, Tanya Lattner wrote: >>> On Jan 28, 2009, at 12:18 PM, David Greene wrote: >>>> I have a buildbot operating to do validating builds of llvm up at >>>> >>>> http://obbligato.org:8080 >>>> >>>> My DSL has been stable enough for the past few months for me to >>>> feel comfortable hosting the buildbot there. >>> >>> We had a discussion in the past on what validate means. Did you ever >>> formalize that? It might be good if you posted (on your website?) >>> what >>> specific criteria you are using to declare a build validated. Or is >>> this just a normal build bot? >> >> We had a long discussion about this. I'll post some information but >> the buildbot essentially does this: >> >> - Build an LLVM without llvm-gcc >> - Run LLVM tests >> - Build llvm-gcc pointing to the newly-build LLVM >> - Rebuild LLVM pointing to the newly-build llvm-gcc >> - Run LLVM tests >> - Run llvm-test >> >> If everything passes for debug, release and paranoid >> (--enable-expensive-checks) we'll consider LLVM validated >> for that target. >> > As I mentioned before, I'm curious what reference point you are using > to determine "pass" for llvm-test. >Here's my idea for this: - The first criteria is "does it compile and run without regressions from the last run?". - The second criteria is "does it run significantly slower than the previous run. - Defining what is "significantly slower" is the hard part here. Some statistical analysis needs to be done on the data; more than just simple ratios. (I'd like to see these analyses done on our nightly tests as well.) They can tell us if: A) the change is significant, and B) if we're gradually regressing over time. The second criteria is *much* more difficult, of course.>>>> It's not yet sending messages to llvmdev. I want to do some more >>>> testing of the setup before I turn it loose on everyone. But you >>>> can >>>> go take a look to see how it operates. >>> >>> I don't think llvm-dev is the right place to be sending mail to. >>> Maybe >>> the testresults list? What mails do you plan to send and how >>> frequent? >> >> The buildbot kicks off every 100 commits or so. There are three >> builds for >> each target (the only buildslaves we have right now are for x86_64- >> linux and >> i686-linux). Each one of those will send an e-mail. >> I'm fine sending it to testresults if people pay attention. I know >> that I >> don't read testresults regularly because there are a lot of test >> runs I >> don't care about. >> > > I still don't think llvm-dev is the right place. We don't want that > mailing list to get cluttered with buildbot test results. You can send > to llvm-test and use filtering if you want to ignore the other > results. >I think that "testresults" is fine for now. That's where the other buildbots send their results. -bw
On Thursday 29 January 2009 03:18, Bill Wendling wrote:> On Jan 28, 2009, at 8:25 PM, Tanya Lattner wrote: > > On Jan 28, 2009, at 3:46 PM, David Greene wrote: > >> On Wednesday 28 January 2009 15:59, Tanya Lattner wrote: > >>> On Jan 28, 2009, at 12:18 PM, David Greene wrote: > >>>> I have a buildbot operating to do validating builds of llvm up at > >>>> > >>>> http://obbligato.org:8080 > >>>> > >>>> My DSL has been stable enough for the past few months for me to > >>>> feel comfortable hosting the buildbot there. > >>> > >>> We had a discussion in the past on what validate means. Did you ever > >>> formalize that? It might be good if you posted (on your website?) > >>> what > >>> specific criteria you are using to declare a build validated. Or is > >>> this just a normal build bot? > >> > >> We had a long discussion about this. I'll post some information but > >> the buildbot essentially does this: > >> > >> - Build an LLVM without llvm-gcc > >> - Run LLVM tests > >> - Build llvm-gcc pointing to the newly-build LLVM > >> - Rebuild LLVM pointing to the newly-build llvm-gcc > >> - Run LLVM tests > >> - Run llvm-test > >> > >> If everything passes for debug, release and paranoid > >> (--enable-expensive-checks) we'll consider LLVM validated > >> for that target. > > > > As I mentioned before, I'm curious what reference point you are using > > to determine "pass" for llvm-test. > > Here's my idea for this: > > - The first criteria is "does it compile and run without regressions > from the last run?". > - The second criteria is "does it run significantly slower than the > previous run.I'm not sure performance regressions should be a requirement for validation. Validation is aobut functional correctness. It gives developers confidence that they can update to a particular revision without breaking anything. Performance regression is important, no doubt, but I'd make it a condition for release, not validation. -Dave
On Thursday 29 January 2009 11:01, David Greene wrote:> > - The first criteria is "does it compile and run without regressions > > from the last run?". > > - The second criteria is "does it run significantly slower than the > > previous run. > > I'm not sure performance regressions should be a requirement for > validation. Validation is aobut functional correctness. It gives > developers confidence that they can update to a particular revision without > breaking anything. > > Performance regression is important, no doubt, but I'd make it a condition > for release, not validation.Thinking about this some more, I could make a case that performance regressions might fit into another type of validation - performance validation. We could have two types of validation: functional and performance. People could individually decide which they care about. The key is to keep things simple for buildbot. We don't want to write a bunch of complex test code in buildbot to do the performance checks / statistical analysis Bill is talking about. Buildbot wants to see PASS / FAIL / XFAIL, etc. So I think we'd need a nice DejaGNU suite that does performance regression testing. To my knowledge, we don't have one yet. So I'm going to leave it to others to develop such a suite and if and when one pops up we can look at integrating it into the buildbot validator. We're all learning here and I know that as we go along we'll tweak the way this works. I'm not worried about that. :) -Dave