Michael Gottesman
2013-Mar-27 22:57 UTC
[LLVMdev] llvmlab (phased buildmaster) is in production mode!
Hello LLVM Dev and Clang Dev! David Dean and I just finished bringing up a new build master on lab.llvm.org, llvmlab, which is located at the url http://lab.llvm.org:8013 and is in #llvm under the username llvmlab. llvmlab is different than the current buildbot based continuous integration systems llvm uses; llvmlab is a phased builder based system. The high level details of the phased builder system are as follows: 1. Builders are grouped together into phases as sub builders. Each phase is a builder itself and triggers said grouped ``sub builders'' and assuming that all of the sub builders complete, triggers its successor phased builder if one exists. This creates a gating like effect. 2. All phases are gated together in a linear fashion, yielding via the linearity the phenomena that if an earlier phase does not succeed, then no later phases run. The key idea here is if we know that there is a fundamental issue with the compiler why try to build 20 compilers, when performing one quick build is all that is needed to ascertain such a fact? Also if we can not build a compiler successfully, why try to do LNT performance runs? This gets rid of pointless work, stops excessive emails from being sent out for 1 bad commit, and reduces cycle time especially if certain builds take significantly longer than others to fail. 3. Later phases do broader, longer lasting testing than earlier phases. Thus the 4 phases we currently have are: a. Phase 1 (sanity): Phase 1 is a quick non-bootstrapped, non-lto compiler build, to check the ``basic sanity'' of the code base and build process. This generally takes 15-20 minutes to complete. b. Phase 2 (living on): Phase 2 builds bootstrapped compilers for the different configurations that can be used for ``living on'', i.e. good enough for common compilation tasks. This is meant to cycle in up to an hour. c. Phase 3 (tree health): Phase 3 runs performance tests, i.e., LNT (which are not live yet) as well as other compiler builds which take a longer amount of time (a full clang build with LTO enabled for instance). d. Phase 4 (validation): Phase 4 runs longer running validation tests. Currently we have nothing in phase 4, but I am sure that will change = p. 4. Builders in later phases rely on outputs from earlier phases. If we are doing performance runs, why should we compile a new compiler for such a performance run? This is duplicated work! Instead the phased build system stores ``artifacts'', i.e., built compilers, in earlier phases and uses them as compilers for builds in later phases. Thus we could have 30 different Phase 3 LNT performance runs with different configurations all using the same compiler artifacts built in phase 2. This significantly deduplicates work yielding a decreased cycle time. As time moves on we will be moving more and more builders to llvmlab including LNT performance builders and a builder which runs the libcxx test suite. Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130327/5708ae87/attachment.html>
Reed Kotler
2013-Mar-28 00:50 UTC
[LLVMdev] llvmlab (phased buildmaster) is in production mode!
Most of the selections, like "console" for example, do not work when I click on them. On 03/27/2013 03:57 PM, Michael Gottesman wrote:> Hello LLVM Dev and Clang Dev! > > David Dean and I just finished bringing up a new build master on > lab.llvm.org <http://lab.llvm.org/>, llvmlab, which is located at the > url http://lab.llvm.org:8013 <http://lab.llvm.org:8013/> and is in #llvm > under the username llvmlab. > > llvmlab is different than the current buildbot based continuous > integration systems llvm uses; llvmlab is a phased builder based system. > The high level details of the phased builder system are as follows: > > 1. Builders are grouped together into phases as sub builders. Each phase > is a builder itself and triggers said grouped ``sub builders'' and > assuming that all of the sub builders complete, triggers its successor > phased builder if one exists. This creates a gating like effect. > > 2. All phases are gated together in a linear fashion, yielding via the > linearity the phenomena that if an earlier phase does not succeed, then > no later phases run. The key idea here is if we know that there is a > fundamental issue with the compiler why try to build 20 compilers, when > performing one quick build is all that is needed to ascertain such a > fact? Also if we can not build a compiler successfully, why try to do > LNT performance runs? This gets rid of pointless work, stops excessive > emails from being sent out for 1 bad commit, and reduces cycle time > especially if certain builds take significantly longer than others to fail. > > 3. Later phases do broader, longer lasting testing than earlier phases. > Thus the 4 phases we currently have are: > > a. Phase 1 (sanity): Phase 1 is a quick non-bootstrapped, > non-lto compiler build, to check the ``basic sanity'' of the code base > and build process. This generally takes 15-20 minutes to complete. > b. Phase 2 (living on): Phase 2 builds bootstrapped compilers for > the different configurations that can be used for ``living on'', i.e. > good enough for common compilation tasks. This is meant to cycle in up > to an hour. > c. Phase 3 (tree health): Phase 3 runs performance tests, i.e., LNT > (which are not live yet) as well as other compiler builds which take a > longer amount of time (a full clang build with LTO enabled for instance). > d. Phase 4 (validation): Phase 4 runs longer running validation tests. > Currently we have nothing in phase 4, but I am sure that will change = p. > > 4. Builders in later phases rely on outputs from earlier phases. If we > are doing performance runs, why should we compile a new compiler for > such a performance run? This is duplicated work! Instead the phased > build system stores ``artifacts'', i.e., built compilers, in earlier > phases and uses them as compilers for builds in later phases. Thus we > could have 30 different Phase 3 LNT performance runs with different > configurations all using the same compiler artifacts built in phase 2. > This significantly deduplicates work yielding a decreased cycle time. > > As time moves on we will be moving more and more builders to llvmlab > including LNT performance builders and a builder which runs the libcxx > test suite. > > Michael > > > > _______________________________________________ > cfe-dev mailing list > cfe-dev-Tmj1lob9twqVc3sceRu5cw at public.gmane.org > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev >
Sean Silva
2013-Mar-28 00:58 UTC
[LLVMdev] [cfe-dev] llvmlab (phased buildmaster) is in production mode!
Cool! This is great news. I feel like this information should be in our documentation somewhere. Could you start a new file ContinuousIntegration.rst and use this content to seed it? This new page would also be a good place to mention some LLVM idiosyncrasies like smooshlab being Apple-internal but still reporting via IRC; these things have not had a good place to be put yet. AFAIK currently our continuous integration infrastructure is mostly community wisdom and besides a small mention of some of the reporting bots on index.rst there is no documentation describing it. -- Sean Silva -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130327/b9a4e141/attachment.html>
Chandler Carruth
2013-Mar-28 01:41 UTC
[LLVMdev] [cfe-dev] llvmlab (phased buildmaster) is in production mode!
On Wed, Mar 27, 2013 at 3:57 PM, Michael Gottesman <mgottesman at apple.com>wrote:> 3. Later phases do broader, longer lasting testing than earlier phases. > Thus the 4 phases we currently have are: > > a. Phase 1 (sanity): Phase 1 is a quick non-bootstrapped, non-lto > compiler build, to check the ``basic sanity'' of the code base and build > process. This generally takes 15-20 minutes to complete. >While most of this sounds great, this one really doesn't. The sanity tests should be able to run *much* faster than 15-20 minutes. Can we prioritize getting an incremental rebuild bot as the sanity phase on reasonably fast hardware? I think it's important to get through the sanity phase in 1-5 minutes so that phase 2 doesn't get soooo many commits piled up on it when someone checks in code with both a miscompile and a tiny small build break (for example). -Chandler -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130327/26a10692/attachment.html>
Michael Gottesman
2013-Mar-28 04:27 UTC
[LLVMdev] [cfe-dev] llvmlab (phased buildmaster) is in production mode!
Smooshlab is going away so the LLVM idiosyncrasy that you mention will be going away soon = p. The whole idea behind bringing up this infrastructure is so that everything that is Apple-internal but COULD be public is public and is on the phased builders. Anything we can't bring out will (of course) stay internal. Also I agree about the documentation. Michael On Mar 27, 2013, at 5:58 PM, Sean Silva <silvas at purdue.edu> wrote:> Cool! This is great news. > > I feel like this information should be in our documentation somewhere. Could you start a new file ContinuousIntegration.rst and use this content to seed it? This new page would also be a good place to mention some LLVM idiosyncrasies like smooshlab being Apple-internal but still reporting via IRC; these things have not had a good place to be put yet. AFAIK currently our continuous integration infrastructure is mostly community wisdom and besides a small mention of some of the reporting bots on index.rst there is no documentation describing it. > > -- Sean Silva
Michael Gottesman
2013-Mar-28 04:39 UTC
[LLVMdev] llvmlab (phased buildmaster) is in production mode!
This is a known issue. I have actually screwed with this before it is just a matter of getting it into zorg in the right manner. On Mar 27, 2013, at 5:50 PM, Reed Kotler <rkotler at mips.com> wrote:> Most of the selections, like "console" for example, do not work when I click on them. > > On 03/27/2013 03:57 PM, Michael Gottesman wrote: >> Hello LLVM Dev and Clang Dev! >> >> David Dean and I just finished bringing up a new build master on >> lab.llvm.org <http://lab.llvm.org/>, llvmlab, which is located at the >> url http://lab.llvm.org:8013 <http://lab.llvm.org:8013/> and is in #llvm >> under the username llvmlab. >> >> llvmlab is different than the current buildbot based continuous >> integration systems llvm uses; llvmlab is a phased builder based system. >> The high level details of the phased builder system are as follows: >> >> 1. Builders are grouped together into phases as sub builders. Each phase >> is a builder itself and triggers said grouped ``sub builders'' and >> assuming that all of the sub builders complete, triggers its successor >> phased builder if one exists. This creates a gating like effect. >> >> 2. All phases are gated together in a linear fashion, yielding via the >> linearity the phenomena that if an earlier phase does not succeed, then >> no later phases run. The key idea here is if we know that there is a >> fundamental issue with the compiler why try to build 20 compilers, when >> performing one quick build is all that is needed to ascertain such a >> fact? Also if we can not build a compiler successfully, why try to do >> LNT performance runs? This gets rid of pointless work, stops excessive >> emails from being sent out for 1 bad commit, and reduces cycle time >> especially if certain builds take significantly longer than others to fail. >> >> 3. Later phases do broader, longer lasting testing than earlier phases. >> Thus the 4 phases we currently have are: >> >> a. Phase 1 (sanity): Phase 1 is a quick non-bootstrapped, >> non-lto compiler build, to check the ``basic sanity'' of the code base >> and build process. This generally takes 15-20 minutes to complete. >> b. Phase 2 (living on): Phase 2 builds bootstrapped compilers for >> the different configurations that can be used for ``living on'', i.e. >> good enough for common compilation tasks. This is meant to cycle in up >> to an hour. >> c. Phase 3 (tree health): Phase 3 runs performance tests, i.e., LNT >> (which are not live yet) as well as other compiler builds which take a >> longer amount of time (a full clang build with LTO enabled for instance). >> d. Phase 4 (validation): Phase 4 runs longer running validation tests. >> Currently we have nothing in phase 4, but I am sure that will change = p. >> >> 4. Builders in later phases rely on outputs from earlier phases. If we >> are doing performance runs, why should we compile a new compiler for >> such a performance run? This is duplicated work! Instead the phased >> build system stores ``artifacts'', i.e., built compilers, in earlier >> phases and uses them as compilers for builds in later phases. Thus we >> could have 30 different Phase 3 LNT performance runs with different >> configurations all using the same compiler artifacts built in phase 2. >> This significantly deduplicates work yielding a decreased cycle time. >> >> As time moves on we will be moving more and more builders to llvmlab >> including LNT performance builders and a builder which runs the libcxx >> test suite. >> >> Michael >> >> >> >> _______________________________________________ >> cfe-dev mailing list >> cfe-dev-Tmj1lob9twqVc3sceRu5cw at public.gmane.org >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev >> > > > _______________________________________________ > LLVM Developers mailing list > LLVMdev at cs.uiuc.edu http://llvm.cs.uiuc.edu > http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
Michael Gottesman
2013-Mar-28 05:09 UTC
[LLVMdev] [cfe-dev] llvmlab (phased buildmaster) is in production mode!
On Mar 27, 2013, at 6:41 PM, Chandler Carruth <chandlerc at google.com> wrote:> > On Wed, Mar 27, 2013 at 3:57 PM, Michael Gottesman <mgottesman at apple.com> wrote: > 3. Later phases do broader, longer lasting testing than earlier phases. Thus the 4 phases we currently have are: > > a. Phase 1 (sanity): Phase 1 is a quick non-bootstrapped, non-lto compiler build, to check the ``basic sanity'' of the code base and build process. This generally takes 15-20 minutes to complete. > > While most of this sounds great, this one really doesn't. > > The sanity tests should be able to run *much* faster than 15-20 minutes.Agreed. I think getting the sanity time down further is possible and very important. IIRC gribozavr has a ninja cmake bot that does clean builds in < 10 minutes. I think that that is a first step (bringing me to your question).> Can we prioritize getting an incremental rebuild bot as the sanity phase on reasonably fast hardware?Incremental builds are quicker but less robust than clean builds. The nice thing about always doing clean builds is that it simplify things by throwing out the potential of any build failures due to a dirty build directory (but maybe I am paranoid). If we want to do incremental builds (which note I am not averse to btw), we should at least set up some manner to clean the build directory if we get a build failure due to a dirty build directory that does not involve sshing into the machine. Perhaps if a builder fails a number of times in a row, clean it? Also, hardware contributions are always welcome = p. Michael -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20130327/7bf388be/attachment.html>
Reasonably Related Threads
- [LLVMdev] llvmlab (phased buildmaster) is in production mode!
- [LLVMdev] [cfe-dev] llvmlab (phased buildmaster) is in production mode!
- [LLVMdev] [cfe-dev] llvmlab (phased buildmaster) is in production mode!
- [LLVMdev] [cfe-dev] llvmlab (phased buildmaster) is in production mode!
- [LLVMdev] [Zorg] Reorganisation, documentation and other issues