On Wednesday 28 January 2009 15:59, Tanya Lattner wrote:> On Jan 28, 2009, at 12:18 PM, David Greene wrote: > > I have a buildbot operating to do validating builds of llvm up at > > > > http://obbligato.org:8080 > > > > My DSL has been stable enough for the past few months for me to > > feel comfortable hosting the buildbot there. > > We had a discussion in the past on what validate means. Did you ever > formalize that? It might be good if you posted (on your website?) what > specific criteria you are using to declare a build validated. Or is > this just a normal build bot?We had a long discussion about this. I'll post some information but the buildbot essentially does this: - Build an LLVM without llvm-gcc - Run LLVM tests - Build llvm-gcc pointing to the newly-build LLVM - Rebuild LLVM pointing to the newly-build llvm-gcc - Run LLVM tests - Run llvm-test If everything passes for debug, release and paranoid (--enable-expensive-checks) we'll consider LLVM validated for that target.> > It's not yet sending messages to llvmdev. I want to do some more > > testing of the setup before I turn it loose on everyone. But you can > > go take a look to see how it operates. > > I don't think llvm-dev is the right place to be sending mail to. Maybe > the testresults list? What mails do you plan to send and how frequent?The buildbot kicks off every 100 commits or so. There are three builds for each target (the only buildslaves we have right now are for x86_64-linux and i686-linux). Each one of those will send an e-mail. I'm fine sending it to testresults if people pay attention. I know that I don't read testresults regularly because there are a lot of test runs I don't care about. The whole point of the validation process is to identify bugs quickly so they get fixed quickly and we keep llvm stable. It means people will have to monitor it and react when stuff doesn't work. -Dave
If you create a slave name and password for me, i'm happy to put one of the ubuntu 8.04 8 core machines i have running it. They are x86_64-linux (you would need to add -j8, which can be done through properties easily) On Thu, Jan 29, 2009 at 10:46 AM, David Greene <dag at cray.com> wrote:> On Wednesday 28 January 2009 15:59, Tanya Lattner wrote: >> On Jan 28, 2009, at 12:18 PM, David Greene wrote: >> > I have a buildbot operating to do validating builds of llvm up at >> > >> > http://obbligato.org:8080 >> > >> > My DSL has been stable enough for the past few months for me to >> > feel comfortable hosting the buildbot there. >> >> We had a discussion in the past on what validate means. Did you ever >> formalize that? It might be good if you posted (on your website?) what >> specific criteria you are using to declare a build validated. Or is >> this just a normal build bot? > > We had a long discussion about this. I'll post some information but > the buildbot essentially does this: > > - Build an LLVM without llvm-gcc > - Run LLVM tests > - Build llvm-gcc pointing to the newly-build LLVM > - Rebuild LLVM pointing to the newly-build llvm-gcc > - Run LLVM tests > - Run llvm-test > > If everything passes for debug, release and paranoid > (--enable-expensive-checks) we'll consider LLVM validated > for that target. > >> > It's not yet sending messages to llvmdev. I want to do some more >> > testing of the setup before I turn it loose on everyone. But you can >> > go take a look to see how it operates. >> >> I don't think llvm-dev is the right place to be sending mail to. Maybe >> the testresults list? What mails do you plan to send and how frequent? > > The buildbot kicks off every 100 commits or so. There are three builds for > each target (the only buildslaves we have right now are for x86_64-linux and > i686-linux). Each one of those will send an e-mail. > > I'm fine sending it to testresults if people pay attention. I know that I > don't read testresults regularly because there are a lot of test runs I > don't care about. > > The whole point of the validation process is to identify bugs quickly so > they get fixed quickly and we keep llvm stable. It means people will > have to monitor it and react when stuff doesn't work. > > -Dave > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >
Also, if you want to put the buildbot status with the other buildbots we have on google1.osuosl.org, i'm happy to add your config to the master On Thu, Jan 29, 2009 at 11:53 AM, Daniel Berlin <dberlin at dberlin.org> wrote:> If you create a slave name and password for me, i'm happy to put one > of the ubuntu 8.04 8 core machines i have running it. They are > x86_64-linux > (you would need to add -j8, which can be done through properties easily) > > On Thu, Jan 29, 2009 at 10:46 AM, David Greene <dag at cray.com> wrote: >> On Wednesday 28 January 2009 15:59, Tanya Lattner wrote: >>> On Jan 28, 2009, at 12:18 PM, David Greene wrote: >>> > I have a buildbot operating to do validating builds of llvm up at >>> > >>> > http://obbligato.org:8080 >>> > >>> > My DSL has been stable enough for the past few months for me to >>> > feel comfortable hosting the buildbot there. >>> >>> We had a discussion in the past on what validate means. Did you ever >>> formalize that? It might be good if you posted (on your website?) what >>> specific criteria you are using to declare a build validated. Or is >>> this just a normal build bot? >> >> We had a long discussion about this. I'll post some information but >> the buildbot essentially does this: >> >> - Build an LLVM without llvm-gcc >> - Run LLVM tests >> - Build llvm-gcc pointing to the newly-build LLVM >> - Rebuild LLVM pointing to the newly-build llvm-gcc >> - Run LLVM tests >> - Run llvm-test >> >> If everything passes for debug, release and paranoid >> (--enable-expensive-checks) we'll consider LLVM validated >> for that target. >> >>> > It's not yet sending messages to llvmdev. I want to do some more >>> > testing of the setup before I turn it loose on everyone. But you can >>> > go take a look to see how it operates. >>> >>> I don't think llvm-dev is the right place to be sending mail to. Maybe >>> the testresults list? What mails do you plan to send and how frequent? >> >> The buildbot kicks off every 100 commits or so. There are three builds for >> each target (the only buildslaves we have right now are for x86_64-linux and >> i686-linux). Each one of those will send an e-mail. >> >> I'm fine sending it to testresults if people pay attention. I know that I >> don't read testresults regularly because there are a lot of test runs I >> don't care about. >> >> The whole point of the validation process is to identify bugs quickly so >> they get fixed quickly and we keep llvm stable. It means people will >> have to monitor it and react when stuff doesn't work. >> >> -Dave >> _______________________________________________ >> LLVM Developers mailing list >> LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu >> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev >> >
On Jan 28, 2009, at 3:46 PM, David Greene wrote:> On Wednesday 28 January 2009 15:59, Tanya Lattner wrote: >> On Jan 28, 2009, at 12:18 PM, David Greene wrote: >>> I have a buildbot operating to do validating builds of llvm up at >>> >>> http://obbligato.org:8080 >>> >>> My DSL has been stable enough for the past few months for me to >>> feel comfortable hosting the buildbot there. >> >> We had a discussion in the past on what validate means. Did you ever >> formalize that? It might be good if you posted (on your website?) >> what >> specific criteria you are using to declare a build validated. Or is >> this just a normal build bot? > > We had a long discussion about this. I'll post some information but > the buildbot essentially does this: > > - Build an LLVM without llvm-gcc > - Run LLVM tests > - Build llvm-gcc pointing to the newly-build LLVM > - Rebuild LLVM pointing to the newly-build llvm-gcc > - Run LLVM tests > - Run llvm-test > > If everything passes for debug, release and paranoid > (--enable-expensive-checks) we'll consider LLVM validated > for that target. >As I mentioned before, I'm curious what reference point you are using to determine "pass" for llvm-test.>>> It's not yet sending messages to llvmdev. I want to do some more >>> testing of the setup before I turn it loose on everyone. But you >>> can >>> go take a look to see how it operates. >> >> I don't think llvm-dev is the right place to be sending mail to. >> Maybe >> the testresults list? What mails do you plan to send and how >> frequent? > > The buildbot kicks off every 100 commits or so. There are three > builds for > each target (the only buildslaves we have right now are for x86_64- > linux and > i686-linux). Each one of those will send an e-mail. > I'm fine sending it to testresults if people pay attention. I know > that I > don't read testresults regularly because there are a lot of test > runs I > don't care about. >I still don't think llvm-dev is the right place. We don't want that mailing list to get cluttered with buildbot test results. You can send to llvm-test and use filtering if you want to ignore the other results. Thanks, -Tanya> The whole point of the validation process is to identify bugs > quickly so > they get fixed quickly and we keep llvm stable. It means people will > have to monitor it and react when stuff doesn't work.> -Dave > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
On Jan 28, 2009, at 8:25 PM, Tanya Lattner wrote:> On Jan 28, 2009, at 3:46 PM, David Greene wrote: > >> On Wednesday 28 January 2009 15:59, Tanya Lattner wrote: >>> On Jan 28, 2009, at 12:18 PM, David Greene wrote: >>>> I have a buildbot operating to do validating builds of llvm up at >>>> >>>> http://obbligato.org:8080 >>>> >>>> My DSL has been stable enough for the past few months for me to >>>> feel comfortable hosting the buildbot there. >>> >>> We had a discussion in the past on what validate means. Did you ever >>> formalize that? It might be good if you posted (on your website?) >>> what >>> specific criteria you are using to declare a build validated. Or is >>> this just a normal build bot? >> >> We had a long discussion about this. I'll post some information but >> the buildbot essentially does this: >> >> - Build an LLVM without llvm-gcc >> - Run LLVM tests >> - Build llvm-gcc pointing to the newly-build LLVM >> - Rebuild LLVM pointing to the newly-build llvm-gcc >> - Run LLVM tests >> - Run llvm-test >> >> If everything passes for debug, release and paranoid >> (--enable-expensive-checks) we'll consider LLVM validated >> for that target. >> > As I mentioned before, I'm curious what reference point you are using > to determine "pass" for llvm-test. >Here's my idea for this: - The first criteria is "does it compile and run without regressions from the last run?". - The second criteria is "does it run significantly slower than the previous run. - Defining what is "significantly slower" is the hard part here. Some statistical analysis needs to be done on the data; more than just simple ratios. (I'd like to see these analyses done on our nightly tests as well.) They can tell us if: A) the change is significant, and B) if we're gradually regressing over time. The second criteria is *much* more difficult, of course.>>>> It's not yet sending messages to llvmdev. I want to do some more >>>> testing of the setup before I turn it loose on everyone. But you >>>> can >>>> go take a look to see how it operates. >>> >>> I don't think llvm-dev is the right place to be sending mail to. >>> Maybe >>> the testresults list? What mails do you plan to send and how >>> frequent? >> >> The buildbot kicks off every 100 commits or so. There are three >> builds for >> each target (the only buildslaves we have right now are for x86_64- >> linux and >> i686-linux). Each one of those will send an e-mail. >> I'm fine sending it to testresults if people pay attention. I know >> that I >> don't read testresults regularly because there are a lot of test >> runs I >> don't care about. >> > > I still don't think llvm-dev is the right place. We don't want that > mailing list to get cluttered with buildbot test results. You can send > to llvm-test and use filtering if you want to ignore the other > results. >I think that "testresults" is fine for now. That's where the other buildbots send their results. -bw